You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by RP Schell <rp...@schellonline.com> on 2014/04/29 19:58:38 UTC
Certificates are switching on threads in the middle of a test run
Hey everyone,
We are attempting to simulate a large user load (thousands) on our
service using JMeter 2.11. To facilitate this, we have created a JKS
keystore containing 4000 unique PKI certificates. We have set the
appropriate options for javax.net.ssl.keyStore and
javax.net.ssl.keyStorePassword and added a Keystore Configuration
component with an alias start index of 0 and alias end index of 3999.
We verified that we can access the system as unique users using JMeter
with this set up.
We started seeing some strange results come back from the JMeter test
runs and determined that the errors returned by our web server were
due to the user certificate switching in the middle of a test run for
a given JMeter thread. More specifically, we added a HTTP request
that gives back some indication of who we are logged in as at the end
of the test run loop. It appears every thread in JMeter is switching
the user it is coming in as about every 5 loops of our test run. This
occurs for each thread whether we set it to run with one or multiple
threads.
This causes problems for our testing as some HTTPS requests assume we
are still the same user and will deny access if the user switches
mid-way through the test run. We verified that
https.sessioncontext.shared=false and
https.use.cached.ssl.context=true is set in the jmeter.properties
file. We did try other combinations of these two options w/o success.
We also tried setting the implementation for HTTP requests to
HTTPClient3.1 as well as HTTPClient4. The behavior is the same.
Is this behavior intended by JMeter? The only way I can prevent the
certificate from switching mid-test is to set the alias start and end
index to the same number in the Keystore Configuration component. Of
course, this is not what we want as it locks all JMeter threads into
running as the exact same user and does not allow proper simulation of
a multi-user load.
Any insight into multi-user certificate loading in JMeter is much
appreciated. Thanks!
-Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Certificates are switching on threads in the middle of a test run
Posted by Philippe Mouawad <ph...@gmail.com>.
On Tue, May 13, 2014 at 5:15 PM, RP Schell <rp...@schellonline.com>wrote:
> Hi Philippe,
>
> Thanks for the reply. Unfortunately, using the CSV method still
> suffers from the certificate switching problem (i.e. a single thread
> changes certs as it executes). The CSV method just changes the order
> in which certificates get used (that of the CSV file).
>
Did you try something like this:
|-CSV DataSet
|-Thread Group (forever)
|--- Loop Controller (forever)
|------Samples here won't switch
>
> If we change Apache's MaxKeepAliveRequests to 0 (unlimited) so that
> connections from Jmeter to Apache never get closed, the certificate
> switching doesn't happen. It seems clear that JMeter wants to change
> certs when its connection to Apache gets closed.
>
Yes this is regular.
>
> -Bill
>
>
> On Wed, May 7, 2014 at 6:30 PM, Philippe Mouawad
> <p....@ubik-ingenierie.com> wrote:
> > Hello,
> > that sounds reasonable, thanks for investigation.
> > Indeed as keepalive close leads to socket close, a new init of ssl
> context
> > occurs leading to an increase in counter and thus leading to the switch.
> >
> > Switching to the method using CSV (as proposed by Ubik Load Pack) and
> > variable name should not suffer from this I think.
> > But a confirmation is welcome.
> >
> > Regards
> >
> >
> >
> > On Wednesday, May 7, 2014, RP Schell <rp...@schellonline.com> wrote:
> >
> >>
> >> We simplified the JMeter plan exhibiting the certificate switching
> >> problem down to one thread doing single request in a loop.
> >> The unwanted certificate switch seems to be related to connection
> >> keep-alive behavior.
> >> We see the certificate change when server closes the connection after
> >> the keep-alive 'max' value decrements down to zero.
> >>
> >> For example, here are (part of) the response headers leading up to
> >> the certificate switch:
> >>
> >> HTTP/1.1 304 Not Modified
> >> Connection: Keep-Alive
> >> Keep-Alive: timeout=10, max=2
> >>
> >> HTTP/1.1 304 Not Modified
> >> Connection: Keep-Alive
> >> Keep-Alive: timeout=10, max=1
> >>
> >> HTTP/1.1 304 Not Modified
> >> Connection: close
> >>
> >> ============ NEXT REQUEST IS ISSUED WITH A DIFFERENT CERT ===========
> >>
> >> HTTP/1.1 200 OK
> >> Keep-Alive: timeout=10, max=500
> >> Connection: Keep-Alive
> >>
> >> On Tue, May 6, 2014 at 6:21 AM, UBIK LOAD PACK Support
> >> <su...@ubikloadpack.com> wrote:
> >> > Hello,
> >> > Can you show the structure of your Test ?
> >> > Maybe open a bug with the Test Plan simplified .
> >> >
> >> > Clearly regarding "https.use.cached.ssl.context to true seems to be do
> >> > better", this does not work as it leads to sharing the certificate
> >> between
> >> > users.
> >> >
> >> > Regards
> >> >
> >> >
> >> > On Tue, May 6, 2014 at 12:34 AM, RP Schell <rpschell@schellonline.com
> >> >wrote:
> >> >
> >> >>
> >> >> Thanks for the feedback!
> >> >>
> >> >> Yup, as stated I have tried https.use.cached.ssl.context=false,
> >> >> HttpClient4, and https.sessioncontext.shared=false without success.
> >> >> Setting https.use.cached.ssl.context to true seems to be do better
> >> >> because the user switching does not happen as frequently, but it
> still
> >> >> happens setting that option to true or false.
> >> >>
> >> >> In answer to your question, we typically put the keystore
> >> >> configuration under the test plan itself at the same level as the
> >> >> thread group that is being defined. The plan itself only defines a
> >> >> single thread group. We have tried putting the keystore
> configuration
> >> >> under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
> >> >> at the top level or under the thread group when there is only one
> >> >> thread group defined has the same behavior.
> >> >>
> >> >> I had not tried using the keystore configuration with a CSV data set
> >> >> configuration. Thanks for that suggestion. I tried an experiment
> >> >> with that. The CSV allows the control of the what certificate is
> used
> >> >> and in what order. However, it does not solve the ultimate problem
> of
> >> >> JMeter switching users. The CSV data set configuration simply causes
> >> >> JMeter to switch users as defined in the order of the CSV instead.
> >> >>
> >> >> I am seeing a pattern emerge from using the CSV though. Our test
> >> >> users are numbered chronologically (e.g., user0, user1, user2, user3,
> >> >> etc). Every 5 loops through the JMeter test, the user identified is
> 5
> >> >> numbers higher than before. For example, the test starts at coming
> in
> >> >> as user 0. On the 5th looping of the test in that thread group, the
> >> >> user is now identified as user 4, and we start getting errors because
> >> >> the server expected us to still be user 0.
> >> >>
> >> >> Again, it looks like JMeter is switching users on me when it restarts
> >> >> the looping of the test plan. How do we control this. It seems none
> >> >> of the options I have mentioned trying allow for it. Thanks!
> >> >>
> >> >> -Bill
> >> >>
> >> >> On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
> >> >> <su...@ubikloadpack.com> wrote:
> >> >> > Hello,
> >> >> > As per:
> >> >> >
> >> >>
> >>
> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
> >> >> >
> >> >> >
> >> >> > https.use.cached.ssl.context must be set to false and httpclient4
> >> must be
> >> >> > used
> >> >> > Don't use:
> >> >> > https.sessioncontext.shared=false
> >> >> >
> >> >> > Also it is much better since 2.11 to use :
> >> >> > Variable name holding certificate aliasVariable name that will
> contain
> >> >> the
> >> >> > alias to use for authentication by client certificate. Variable
> value
> >> >> will
> >> >> > be filled from CSV Data Set for example. In the screenshot,
> >> >> > "certificat_ssl" will also be a variable in CSV Data Set.False
> >> >> > Read doc on start and end index to be sure you configure them
> >> correctly
> >> >> >
> >> >> > You speak about different thread groups, where do you put keystore
> >> >> config ?
> >> >> >
> >> >> > Regards
> >> >> > @ubikloadpack
> >> >> > On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com>
> >> wrote:
> >> >> >
> >> >> >> Hey everyone,
> >> >> >>
> >> >> >> We are attempting to simulate a large user
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> > Ubik-Ingénierie
> >
> > UBIK LOAD PACK Web Site <http://www.ubikloadpack.com/>
> >
> > UBIK LOAD PACK on TWITTER <https://twitter.com/ubikloadpack>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
--
Cordialement.
Philippe Mouawad.
Re: Certificates are switching on threads in the middle of a test run
Posted by RP Schell <rp...@schellonline.com>.
Hi Philippe,
Thanks for the reply. Unfortunately, using the CSV method still
suffers from the certificate switching problem (i.e. a single thread
changes certs as it executes). The CSV method just changes the order
in which certificates get used (that of the CSV file).
If we change Apache's MaxKeepAliveRequests to 0 (unlimited) so that
connections from Jmeter to Apache never get closed, the certificate
switching doesn't happen. It seems clear that JMeter wants to change
certs when its connection to Apache gets closed.
-Bill
On Wed, May 7, 2014 at 6:30 PM, Philippe Mouawad
<p....@ubik-ingenierie.com> wrote:
> Hello,
> that sounds reasonable, thanks for investigation.
> Indeed as keepalive close leads to socket close, a new init of ssl context
> occurs leading to an increase in counter and thus leading to the switch.
>
> Switching to the method using CSV (as proposed by Ubik Load Pack) and
> variable name should not suffer from this I think.
> But a confirmation is welcome.
>
> Regards
>
>
>
> On Wednesday, May 7, 2014, RP Schell <rp...@schellonline.com> wrote:
>
>>
>> We simplified the JMeter plan exhibiting the certificate switching
>> problem down to one thread doing single request in a loop.
>> The unwanted certificate switch seems to be related to connection
>> keep-alive behavior.
>> We see the certificate change when server closes the connection after
>> the keep-alive 'max' value decrements down to zero.
>>
>> For example, here are (part of) the response headers leading up to
>> the certificate switch:
>>
>> HTTP/1.1 304 Not Modified
>> Connection: Keep-Alive
>> Keep-Alive: timeout=10, max=2
>>
>> HTTP/1.1 304 Not Modified
>> Connection: Keep-Alive
>> Keep-Alive: timeout=10, max=1
>>
>> HTTP/1.1 304 Not Modified
>> Connection: close
>>
>> ============ NEXT REQUEST IS ISSUED WITH A DIFFERENT CERT ===========
>>
>> HTTP/1.1 200 OK
>> Keep-Alive: timeout=10, max=500
>> Connection: Keep-Alive
>>
>> On Tue, May 6, 2014 at 6:21 AM, UBIK LOAD PACK Support
>> <su...@ubikloadpack.com> wrote:
>> > Hello,
>> > Can you show the structure of your Test ?
>> > Maybe open a bug with the Test Plan simplified .
>> >
>> > Clearly regarding "https.use.cached.ssl.context to true seems to be do
>> > better", this does not work as it leads to sharing the certificate
>> between
>> > users.
>> >
>> > Regards
>> >
>> >
>> > On Tue, May 6, 2014 at 12:34 AM, RP Schell <rpschell@schellonline.com
>> >wrote:
>> >
>> >>
>> >> Thanks for the feedback!
>> >>
>> >> Yup, as stated I have tried https.use.cached.ssl.context=false,
>> >> HttpClient4, and https.sessioncontext.shared=false without success.
>> >> Setting https.use.cached.ssl.context to true seems to be do better
>> >> because the user switching does not happen as frequently, but it still
>> >> happens setting that option to true or false.
>> >>
>> >> In answer to your question, we typically put the keystore
>> >> configuration under the test plan itself at the same level as the
>> >> thread group that is being defined. The plan itself only defines a
>> >> single thread group. We have tried putting the keystore configuration
>> >> under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
>> >> at the top level or under the thread group when there is only one
>> >> thread group defined has the same behavior.
>> >>
>> >> I had not tried using the keystore configuration with a CSV data set
>> >> configuration. Thanks for that suggestion. I tried an experiment
>> >> with that. The CSV allows the control of the what certificate is used
>> >> and in what order. However, it does not solve the ultimate problem of
>> >> JMeter switching users. The CSV data set configuration simply causes
>> >> JMeter to switch users as defined in the order of the CSV instead.
>> >>
>> >> I am seeing a pattern emerge from using the CSV though. Our test
>> >> users are numbered chronologically (e.g., user0, user1, user2, user3,
>> >> etc). Every 5 loops through the JMeter test, the user identified is 5
>> >> numbers higher than before. For example, the test starts at coming in
>> >> as user 0. On the 5th looping of the test in that thread group, the
>> >> user is now identified as user 4, and we start getting errors because
>> >> the server expected us to still be user 0.
>> >>
>> >> Again, it looks like JMeter is switching users on me when it restarts
>> >> the looping of the test plan. How do we control this. It seems none
>> >> of the options I have mentioned trying allow for it. Thanks!
>> >>
>> >> -Bill
>> >>
>> >> On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
>> >> <su...@ubikloadpack.com> wrote:
>> >> > Hello,
>> >> > As per:
>> >> >
>> >>
>> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
>> >> >
>> >> >
>> >> > https.use.cached.ssl.context must be set to false and httpclient4
>> must be
>> >> > used
>> >> > Don't use:
>> >> > https.sessioncontext.shared=false
>> >> >
>> >> > Also it is much better since 2.11 to use :
>> >> > Variable name holding certificate aliasVariable name that will contain
>> >> the
>> >> > alias to use for authentication by client certificate. Variable value
>> >> will
>> >> > be filled from CSV Data Set for example. In the screenshot,
>> >> > "certificat_ssl" will also be a variable in CSV Data Set.False
>> >> > Read doc on start and end index to be sure you configure them
>> correctly
>> >> >
>> >> > You speak about different thread groups, where do you put keystore
>> >> config ?
>> >> >
>> >> > Regards
>> >> > @ubikloadpack
>> >> > On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com>
>> wrote:
>> >> >
>> >> >> Hey everyone,
>> >> >>
>> >> >> We are attempting to simulate a large user
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
> Ubik-Ingénierie
>
> UBIK LOAD PACK Web Site <http://www.ubikloadpack.com/>
>
> UBIK LOAD PACK on TWITTER <https://twitter.com/ubikloadpack>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Certificates are switching on threads in the middle of a test run
Posted by Philippe Mouawad <p....@ubik-ingenierie.com>.
Hello,
that sounds reasonable, thanks for investigation.
Indeed as keepalive close leads to socket close, a new init of ssl context
occurs leading to an increase in counter and thus leading to the switch.
Switching to the method using CSV (as proposed by Ubik Load Pack) and
variable name should not suffer from this I think.
But a confirmation is welcome.
Regards
On Wednesday, May 7, 2014, RP Schell <rp...@schellonline.com> wrote:
>
> We simplified the JMeter plan exhibiting the certificate switching
> problem down to one thread doing single request in a loop.
> The unwanted certificate switch seems to be related to connection
> keep-alive behavior.
> We see the certificate change when server closes the connection after
> the keep-alive 'max' value decrements down to zero.
>
> For example, here are (part of) the response headers leading up to
> the certificate switch:
>
> HTTP/1.1 304 Not Modified
> Connection: Keep-Alive
> Keep-Alive: timeout=10, max=2
>
> HTTP/1.1 304 Not Modified
> Connection: Keep-Alive
> Keep-Alive: timeout=10, max=1
>
> HTTP/1.1 304 Not Modified
> Connection: close
>
> ============ NEXT REQUEST IS ISSUED WITH A DIFFERENT CERT ===========
>
> HTTP/1.1 200 OK
> Keep-Alive: timeout=10, max=500
> Connection: Keep-Alive
>
> On Tue, May 6, 2014 at 6:21 AM, UBIK LOAD PACK Support
> <su...@ubikloadpack.com> wrote:
> > Hello,
> > Can you show the structure of your Test ?
> > Maybe open a bug with the Test Plan simplified .
> >
> > Clearly regarding "https.use.cached.ssl.context to true seems to be do
> > better", this does not work as it leads to sharing the certificate
> between
> > users.
> >
> > Regards
> >
> >
> > On Tue, May 6, 2014 at 12:34 AM, RP Schell <rpschell@schellonline.com
> >wrote:
> >
> >>
> >> Thanks for the feedback!
> >>
> >> Yup, as stated I have tried https.use.cached.ssl.context=false,
> >> HttpClient4, and https.sessioncontext.shared=false without success.
> >> Setting https.use.cached.ssl.context to true seems to be do better
> >> because the user switching does not happen as frequently, but it still
> >> happens setting that option to true or false.
> >>
> >> In answer to your question, we typically put the keystore
> >> configuration under the test plan itself at the same level as the
> >> thread group that is being defined. The plan itself only defines a
> >> single thread group. We have tried putting the keystore configuration
> >> under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
> >> at the top level or under the thread group when there is only one
> >> thread group defined has the same behavior.
> >>
> >> I had not tried using the keystore configuration with a CSV data set
> >> configuration. Thanks for that suggestion. I tried an experiment
> >> with that. The CSV allows the control of the what certificate is used
> >> and in what order. However, it does not solve the ultimate problem of
> >> JMeter switching users. The CSV data set configuration simply causes
> >> JMeter to switch users as defined in the order of the CSV instead.
> >>
> >> I am seeing a pattern emerge from using the CSV though. Our test
> >> users are numbered chronologically (e.g., user0, user1, user2, user3,
> >> etc). Every 5 loops through the JMeter test, the user identified is 5
> >> numbers higher than before. For example, the test starts at coming in
> >> as user 0. On the 5th looping of the test in that thread group, the
> >> user is now identified as user 4, and we start getting errors because
> >> the server expected us to still be user 0.
> >>
> >> Again, it looks like JMeter is switching users on me when it restarts
> >> the looping of the test plan. How do we control this. It seems none
> >> of the options I have mentioned trying allow for it. Thanks!
> >>
> >> -Bill
> >>
> >> On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
> >> <su...@ubikloadpack.com> wrote:
> >> > Hello,
> >> > As per:
> >> >
> >>
> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
> >> >
> >> >
> >> > https.use.cached.ssl.context must be set to false and httpclient4
> must be
> >> > used
> >> > Don't use:
> >> > https.sessioncontext.shared=false
> >> >
> >> > Also it is much better since 2.11 to use :
> >> > Variable name holding certificate aliasVariable name that will contain
> >> the
> >> > alias to use for authentication by client certificate. Variable value
> >> will
> >> > be filled from CSV Data Set for example. In the screenshot,
> >> > "certificat_ssl" will also be a variable in CSV Data Set.False
> >> > Read doc on start and end index to be sure you configure them
> correctly
> >> >
> >> > You speak about different thread groups, where do you put keystore
> >> config ?
> >> >
> >> > Regards
> >> > @ubikloadpack
> >> > On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com>
> wrote:
> >> >
> >> >> Hey everyone,
> >> >>
> >> >> We are attempting to simulate a large user
--
Cordialement.
Philippe Mouawad.
Ubik-Ingénierie
UBIK LOAD PACK Web Site <http://www.ubikloadpack.com/>
UBIK LOAD PACK on TWITTER <https://twitter.com/ubikloadpack>
Re: Certificates are switching on threads in the middle of a test run
Posted by RP Schell <rp...@schellonline.com>.
We simplified the JMeter plan exhibiting the certificate switching
problem down to one thread doing single request in a loop.
The unwanted certificate switch seems to be related to connection
keep-alive behavior.
We see the certificate change when server closes the connection after
the keep-alive 'max' value decrements down to zero.
For example, here are (part of) the response headers leading up to
the certificate switch:
HTTP/1.1 304 Not Modified
Connection: Keep-Alive
Keep-Alive: timeout=10, max=2
HTTP/1.1 304 Not Modified
Connection: Keep-Alive
Keep-Alive: timeout=10, max=1
HTTP/1.1 304 Not Modified
Connection: close
============ NEXT REQUEST IS ISSUED WITH A DIFFERENT CERT ===========
HTTP/1.1 200 OK
Keep-Alive: timeout=10, max=500
Connection: Keep-Alive
On Tue, May 6, 2014 at 6:21 AM, UBIK LOAD PACK Support
<su...@ubikloadpack.com> wrote:
> Hello,
> Can you show the structure of your Test ?
> Maybe open a bug with the Test Plan simplified .
>
> Clearly regarding "https.use.cached.ssl.context to true seems to be do
> better", this does not work as it leads to sharing the certificate between
> users.
>
> Regards
>
>
> On Tue, May 6, 2014 at 12:34 AM, RP Schell <rp...@schellonline.com>wrote:
>
>>
>> Thanks for the feedback!
>>
>> Yup, as stated I have tried https.use.cached.ssl.context=false,
>> HttpClient4, and https.sessioncontext.shared=false without success.
>> Setting https.use.cached.ssl.context to true seems to be do better
>> because the user switching does not happen as frequently, but it still
>> happens setting that option to true or false.
>>
>> In answer to your question, we typically put the keystore
>> configuration under the test plan itself at the same level as the
>> thread group that is being defined. The plan itself only defines a
>> single thread group. We have tried putting the keystore configuration
>> under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
>> at the top level or under the thread group when there is only one
>> thread group defined has the same behavior.
>>
>> I had not tried using the keystore configuration with a CSV data set
>> configuration. Thanks for that suggestion. I tried an experiment
>> with that. The CSV allows the control of the what certificate is used
>> and in what order. However, it does not solve the ultimate problem of
>> JMeter switching users. The CSV data set configuration simply causes
>> JMeter to switch users as defined in the order of the CSV instead.
>>
>> I am seeing a pattern emerge from using the CSV though. Our test
>> users are numbered chronologically (e.g., user0, user1, user2, user3,
>> etc). Every 5 loops through the JMeter test, the user identified is 5
>> numbers higher than before. For example, the test starts at coming in
>> as user 0. On the 5th looping of the test in that thread group, the
>> user is now identified as user 4, and we start getting errors because
>> the server expected us to still be user 0.
>>
>> Again, it looks like JMeter is switching users on me when it restarts
>> the looping of the test plan. How do we control this. It seems none
>> of the options I have mentioned trying allow for it. Thanks!
>>
>> -Bill
>>
>> On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
>> <su...@ubikloadpack.com> wrote:
>> > Hello,
>> > As per:
>> >
>> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
>> >
>> >
>> > https.use.cached.ssl.context must be set to false and httpclient4 must be
>> > used
>> > Don't use:
>> > https.sessioncontext.shared=false
>> >
>> > Also it is much better since 2.11 to use :
>> > Variable name holding certificate aliasVariable name that will contain
>> the
>> > alias to use for authentication by client certificate. Variable value
>> will
>> > be filled from CSV Data Set for example. In the screenshot,
>> > "certificat_ssl" will also be a variable in CSV Data Set.False
>> > Read doc on start and end index to be sure you configure them correctly
>> >
>> > You speak about different thread groups, where do you put keystore
>> config ?
>> >
>> > Regards
>> > @ubikloadpack
>> > On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com> wrote:
>> >
>> >> Hey everyone,
>> >>
>> >> We are attempting to simulate a large user load (thousands) on our
>> >> service using JMeter 2.11. To facilitate this, we have created a JKS
>> >> keystore containing 4000 unique PKI certificates. We have set the
>> >> appropriate options for javax.net.ssl.keyStore and
>> >> javax.net.ssl.keyStorePassword and added a Keystore Configuration
>> >> component with an alias start index of 0 and alias end index of 3999.
>> >> We verified that we can access the system as unique users using JMeter
>> >> with this set up.
>> >>
>> >> We started seeing some strange results come back from the JMeter test
>> >> runs and determined that the errors returned by our web server were
>> >> due to the user certificate switching in the middle of a test run for
>> >> a given JMeter thread. More specifically, we added a HTTP request
>> >> that gives back some indication of who we are logged in as at the end
>> >> of the test run loop. It appears every thread in JMeter is switching
>> >> the user it is coming in as about every 5 loops of our test run. This
>> >> occurs for each thread whether we set it to run with one or multiple
>> >> threads.
>> >>
>> >> This causes problems for our testing as some HTTPS requests assume we
>> >> are still the same user and will deny access if the user switches
>> >> mid-way through the test run. We verified that
>> >> https.sessioncontext.shared=false and
>> >> https.use.cached.ssl.context=true is set in the jmeter.properties
>> >> file. We did try other combinations of these two options w/o success.
>> >>
>> >> We also tried setting the implementation for HTTP requests to
>> >> HTTPClient3.1 as well as HTTPClient4. The behavior is the same.
>> >>
>> >> Is this behavior intended by JMeter? The only way I can prevent the
>> >> certificate from switching mid-test is to set the alias start and end
>> >> index to the same number in the Keystore Configuration component. Of
>> >> course, this is not what we want as it locks all JMeter threads into
>> >> running as the exact same user and does not allow proper simulation of
>> >> a multi-user load.
>> >>
>> >> Any insight into multi-user certificate loading in JMeter is much
>> >> appreciated. Thanks!
>> >>
>> >> -Bill
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org<javascript:;>
>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>> <javascript:;>
>> >>
>> >>
>> >
>> > --
>> >
>> > Regards
>> > Ubik Load Pack <http://ubikloadpack.com> Team
>> > Follow us on Twitter <http://twitter.com/ubikloadpack>
>> >
>> >
>> > Cordialement
>> > L'équipe Ubik Load Pack <http://ubikloadpack.com>
>> > Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
>>
>
>
>
> --
>
> Regards
> Ubik Load Pack <http://ubikloadpack.com> Team
> Follow us on Twitter <http://twitter.com/ubikloadpack>
>
>
> Cordialement
> L'équipe Ubik Load Pack <http://ubikloadpack.com>
> Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Certificates are switching on threads in the middle of a test run
Posted by UBIK LOAD PACK Support <su...@ubikloadpack.com>.
Hello,
Can you show the structure of your Test ?
Maybe open a bug with the Test Plan simplified .
Clearly regarding "https.use.cached.ssl.context to true seems to be do
better", this does not work as it leads to sharing the certificate between
users.
Regards
On Tue, May 6, 2014 at 12:34 AM, RP Schell <rp...@schellonline.com>wrote:
>
> Thanks for the feedback!
>
> Yup, as stated I have tried https.use.cached.ssl.context=false,
> HttpClient4, and https.sessioncontext.shared=false without success.
> Setting https.use.cached.ssl.context to true seems to be do better
> because the user switching does not happen as frequently, but it still
> happens setting that option to true or false.
>
> In answer to your question, we typically put the keystore
> configuration under the test plan itself at the same level as the
> thread group that is being defined. The plan itself only defines a
> single thread group. We have tried putting the keystore configuration
> under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
> at the top level or under the thread group when there is only one
> thread group defined has the same behavior.
>
> I had not tried using the keystore configuration with a CSV data set
> configuration. Thanks for that suggestion. I tried an experiment
> with that. The CSV allows the control of the what certificate is used
> and in what order. However, it does not solve the ultimate problem of
> JMeter switching users. The CSV data set configuration simply causes
> JMeter to switch users as defined in the order of the CSV instead.
>
> I am seeing a pattern emerge from using the CSV though. Our test
> users are numbered chronologically (e.g., user0, user1, user2, user3,
> etc). Every 5 loops through the JMeter test, the user identified is 5
> numbers higher than before. For example, the test starts at coming in
> as user 0. On the 5th looping of the test in that thread group, the
> user is now identified as user 4, and we start getting errors because
> the server expected us to still be user 0.
>
> Again, it looks like JMeter is switching users on me when it restarts
> the looping of the test plan. How do we control this. It seems none
> of the options I have mentioned trying allow for it. Thanks!
>
> -Bill
>
> On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
> <su...@ubikloadpack.com> wrote:
> > Hello,
> > As per:
> >
> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
> >
> >
> > https.use.cached.ssl.context must be set to false and httpclient4 must be
> > used
> > Don't use:
> > https.sessioncontext.shared=false
> >
> > Also it is much better since 2.11 to use :
> > Variable name holding certificate aliasVariable name that will contain
> the
> > alias to use for authentication by client certificate. Variable value
> will
> > be filled from CSV Data Set for example. In the screenshot,
> > "certificat_ssl" will also be a variable in CSV Data Set.False
> > Read doc on start and end index to be sure you configure them correctly
> >
> > You speak about different thread groups, where do you put keystore
> config ?
> >
> > Regards
> > @ubikloadpack
> > On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com> wrote:
> >
> >> Hey everyone,
> >>
> >> We are attempting to simulate a large user load (thousands) on our
> >> service using JMeter 2.11. To facilitate this, we have created a JKS
> >> keystore containing 4000 unique PKI certificates. We have set the
> >> appropriate options for javax.net.ssl.keyStore and
> >> javax.net.ssl.keyStorePassword and added a Keystore Configuration
> >> component with an alias start index of 0 and alias end index of 3999.
> >> We verified that we can access the system as unique users using JMeter
> >> with this set up.
> >>
> >> We started seeing some strange results come back from the JMeter test
> >> runs and determined that the errors returned by our web server were
> >> due to the user certificate switching in the middle of a test run for
> >> a given JMeter thread. More specifically, we added a HTTP request
> >> that gives back some indication of who we are logged in as at the end
> >> of the test run loop. It appears every thread in JMeter is switching
> >> the user it is coming in as about every 5 loops of our test run. This
> >> occurs for each thread whether we set it to run with one or multiple
> >> threads.
> >>
> >> This causes problems for our testing as some HTTPS requests assume we
> >> are still the same user and will deny access if the user switches
> >> mid-way through the test run. We verified that
> >> https.sessioncontext.shared=false and
> >> https.use.cached.ssl.context=true is set in the jmeter.properties
> >> file. We did try other combinations of these two options w/o success.
> >>
> >> We also tried setting the implementation for HTTP requests to
> >> HTTPClient3.1 as well as HTTPClient4. The behavior is the same.
> >>
> >> Is this behavior intended by JMeter? The only way I can prevent the
> >> certificate from switching mid-test is to set the alias start and end
> >> index to the same number in the Keystore Configuration component. Of
> >> course, this is not what we want as it locks all JMeter threads into
> >> running as the exact same user and does not allow proper simulation of
> >> a multi-user load.
> >>
> >> Any insight into multi-user certificate loading in JMeter is much
> >> appreciated. Thanks!
> >>
> >> -Bill
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org<javascript:;>
> >> For additional commands, e-mail: user-help@jmeter.apache.org
> <javascript:;>
> >>
> >>
> >
> > --
> >
> > Regards
> > Ubik Load Pack <http://ubikloadpack.com> Team
> > Follow us on Twitter <http://twitter.com/ubikloadpack>
> >
> >
> > Cordialement
> > L'équipe Ubik Load Pack <http://ubikloadpack.com>
> > Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
>
--
Regards
Ubik Load Pack <http://ubikloadpack.com> Team
Follow us on Twitter <http://twitter.com/ubikloadpack>
Cordialement
L'équipe Ubik Load Pack <http://ubikloadpack.com>
Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
Re: Certificates are switching on threads in the middle of a test run
Posted by Shmuel Krakower <sh...@gmail.com>.
If you use a cookies manager make sure you don't clear cookies on every
iteration.
www.beatsoo.org - free application performance monitoring from world wide
locations.
On May 6, 2014 1:34 AM, "RP Schell" <rp...@schellonline.com> wrote:
> Thanks for the feedback!
>
> Yup, as stated I have tried https.use.cached.ssl.context=false,
> HttpClient4, and https.sessioncontext.shared=false without success.
> Setting https.use.cached.ssl.context to true seems to be do better
> because the user switching does not happen as frequently, but it still
> happens setting that option to true or false.
>
> In answer to your question, we typically put the keystore
> configuration under the test plan itself at the same level as the
> thread group that is being defined. The plan itself only defines a
> single thread group. We have tried putting the keystore configuration
> under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
> at the top level or under the thread group when there is only one
> thread group defined has the same behavior.
>
> I had not tried using the keystore configuration with a CSV data set
> configuration. Thanks for that suggestion. I tried an experiment
> with that. The CSV allows the control of the what certificate is used
> and in what order. However, it does not solve the ultimate problem of
> JMeter switching users. The CSV data set configuration simply causes
> JMeter to switch users as defined in the order of the CSV instead.
>
> I am seeing a pattern emerge from using the CSV though. Our test
> users are numbered chronologically (e.g., user0, user1, user2, user3,
> etc). Every 5 loops through the JMeter test, the user identified is 5
> numbers higher than before. For example, the test starts at coming in
> as user 0. On the 5th looping of the test in that thread group, the
> user is now identified as user 4, and we start getting errors because
> the server expected us to still be user 0.
>
> Again, it looks like JMeter is switching users on me when it restarts
> the looping of the test plan. How do we control this. It seems none
> of the options I have mentioned trying allow for it. Thanks!
>
> -Bill
>
> On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
> <su...@ubikloadpack.com> wrote:
> > Hello,
> > As per:
> >
> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
> >
> >
> > https.use.cached.ssl.context must be set to false and httpclient4 must be
> > used
> > Don't use:
> > https.sessioncontext.shared=false
> >
> > Also it is much better since 2.11 to use :
> > Variable name holding certificate aliasVariable name that will contain
> the
> > alias to use for authentication by client certificate. Variable value
> will
> > be filled from CSV Data Set for example. In the screenshot,
> > "certificat_ssl" will also be a variable in CSV Data Set.False
> > Read doc on start and end index to be sure you configure them correctly
> >
> > You speak about different thread groups, where do you put keystore
> config ?
> >
> > Regards
> > @ubikloadpack
> > On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com> wrote:
> >
> >> Hey everyone,
> >>
> >> We are attempting to simulate a large user load (thousands) on our
> >> service using JMeter 2.11. To facilitate this, we have created a JKS
> >> keystore containing 4000 unique PKI certificates. We have set the
> >> appropriate options for javax.net.ssl.keyStore and
> >> javax.net.ssl.keyStorePassword and added a Keystore Configuration
> >> component with an alias start index of 0 and alias end index of 3999.
> >> We verified that we can access the system as unique users using JMeter
> >> with this set up.
> >>
> >> We started seeing some strange results come back from the JMeter test
> >> runs and determined that the errors returned by our web server were
> >> due to the user certificate switching in the middle of a test run for
> >> a given JMeter thread. More specifically, we added a HTTP request
> >> that gives back some indication of who we are logged in as at the end
> >> of the test run loop. It appears every thread in JMeter is switching
> >> the user it is coming in as about every 5 loops of our test run. This
> >> occurs for each thread whether we set it to run with one or multiple
> >> threads.
> >>
> >> This causes problems for our testing as some HTTPS requests assume we
> >> are still the same user and will deny access if the user switches
> >> mid-way through the test run. We verified that
> >> https.sessioncontext.shared=false and
> >> https.use.cached.ssl.context=true is set in the jmeter.properties
> >> file. We did try other combinations of these two options w/o success.
> >>
> >> We also tried setting the implementation for HTTP requests to
> >> HTTPClient3.1 as well as HTTPClient4. The behavior is the same.
> >>
> >> Is this behavior intended by JMeter? The only way I can prevent the
> >> certificate from switching mid-test is to set the alias start and end
> >> index to the same number in the Keystore Configuration component. Of
> >> course, this is not what we want as it locks all JMeter threads into
> >> running as the exact same user and does not allow proper simulation of
> >> a multi-user load.
> >>
> >> Any insight into multi-user certificate loading in JMeter is much
> >> appreciated. Thanks!
> >>
> >> -Bill
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org<javascript:;>
> >> For additional commands, e-mail: user-help@jmeter.apache.org
> <javascript:;>
> >>
> >>
> >
> > --
> >
> > Regards
> > Ubik Load Pack <http://ubikloadpack.com> Team
> > Follow us on Twitter <http://twitter.com/ubikloadpack>
> >
> >
> > Cordialement
> > L'équipe Ubik Load Pack <http://ubikloadpack.com>
> > Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
Re: Certificates are switching on threads in the middle of a test run
Posted by RP Schell <rp...@schellonline.com>.
Thanks for the feedback!
Yup, as stated I have tried https.use.cached.ssl.context=false,
HttpClient4, and https.sessioncontext.shared=false without success.
Setting https.use.cached.ssl.context to true seems to be do better
because the user switching does not happen as frequently, but it still
happens setting that option to true or false.
In answer to your question, we typically put the keystore
configuration under the test plan itself at the same level as the
thread group that is being defined. The plan itself only defines a
single thread group. We have tried putting the keystore configuration
under the thread group itself. THIS MAKES NO DIFFERENCE. Putting it
at the top level or under the thread group when there is only one
thread group defined has the same behavior.
I had not tried using the keystore configuration with a CSV data set
configuration. Thanks for that suggestion. I tried an experiment
with that. The CSV allows the control of the what certificate is used
and in what order. However, it does not solve the ultimate problem of
JMeter switching users. The CSV data set configuration simply causes
JMeter to switch users as defined in the order of the CSV instead.
I am seeing a pattern emerge from using the CSV though. Our test
users are numbered chronologically (e.g., user0, user1, user2, user3,
etc). Every 5 loops through the JMeter test, the user identified is 5
numbers higher than before. For example, the test starts at coming in
as user 0. On the 5th looping of the test in that thread group, the
user is now identified as user 4, and we start getting errors because
the server expected us to still be user 0.
Again, it looks like JMeter is switching users on me when it restarts
the looping of the test plan. How do we control this. It seems none
of the options I have mentioned trying allow for it. Thanks!
-Bill
On Tue, Apr 29, 2014 at 4:12 PM, UBIK LOAD PACK Support
<su...@ubikloadpack.com> wrote:
> Hello,
> As per:
> http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
>
>
> https.use.cached.ssl.context must be set to false and httpclient4 must be
> used
> Don't use:
> https.sessioncontext.shared=false
>
> Also it is much better since 2.11 to use :
> Variable name holding certificate aliasVariable name that will contain the
> alias to use for authentication by client certificate. Variable value will
> be filled from CSV Data Set for example. In the screenshot,
> "certificat_ssl" will also be a variable in CSV Data Set.False
> Read doc on start and end index to be sure you configure them correctly
>
> You speak about different thread groups, where do you put keystore config ?
>
> Regards
> @ubikloadpack
> On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com> wrote:
>
>> Hey everyone,
>>
>> We are attempting to simulate a large user load (thousands) on our
>> service using JMeter 2.11. To facilitate this, we have created a JKS
>> keystore containing 4000 unique PKI certificates. We have set the
>> appropriate options for javax.net.ssl.keyStore and
>> javax.net.ssl.keyStorePassword and added a Keystore Configuration
>> component with an alias start index of 0 and alias end index of 3999.
>> We verified that we can access the system as unique users using JMeter
>> with this set up.
>>
>> We started seeing some strange results come back from the JMeter test
>> runs and determined that the errors returned by our web server were
>> due to the user certificate switching in the middle of a test run for
>> a given JMeter thread. More specifically, we added a HTTP request
>> that gives back some indication of who we are logged in as at the end
>> of the test run loop. It appears every thread in JMeter is switching
>> the user it is coming in as about every 5 loops of our test run. This
>> occurs for each thread whether we set it to run with one or multiple
>> threads.
>>
>> This causes problems for our testing as some HTTPS requests assume we
>> are still the same user and will deny access if the user switches
>> mid-way through the test run. We verified that
>> https.sessioncontext.shared=false and
>> https.use.cached.ssl.context=true is set in the jmeter.properties
>> file. We did try other combinations of these two options w/o success.
>>
>> We also tried setting the implementation for HTTP requests to
>> HTTPClient3.1 as well as HTTPClient4. The behavior is the same.
>>
>> Is this behavior intended by JMeter? The only way I can prevent the
>> certificate from switching mid-test is to set the alias start and end
>> index to the same number in the Keystore Configuration component. Of
>> course, this is not what we want as it locks all JMeter threads into
>> running as the exact same user and does not allow proper simulation of
>> a multi-user load.
>>
>> Any insight into multi-user certificate loading in JMeter is much
>> appreciated. Thanks!
>>
>> -Bill
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org <javascript:;>
>> For additional commands, e-mail: user-help@jmeter.apache.org<javascript:;>
>>
>>
>
> --
>
> Regards
> Ubik Load Pack <http://ubikloadpack.com> Team
> Follow us on Twitter <http://twitter.com/ubikloadpack>
>
>
> Cordialement
> L'équipe Ubik Load Pack <http://ubikloadpack.com>
> Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Certificates are switching on threads in the middle of a test run
Posted by UBIK LOAD PACK Support <su...@ubikloadpack.com>.
Hello,
As per:
http://jmeter.apache.org/usermanual/component_reference.html#Keystore_Configuration
https.use.cached.ssl.context must be set to false and httpclient4 must be
used
Don't use:
https.sessioncontext.shared=false
Also it is much better since 2.11 to use :
Variable name holding certificate aliasVariable name that will contain the
alias to use for authentication by client certificate. Variable value will
be filled from CSV Data Set for example. In the screenshot,
"certificat_ssl" will also be a variable in CSV Data Set.False
Read doc on start and end index to be sure you configure them correctly
You speak about different thread groups, where do you put keystore config ?
Regards
@ubikloadpack
On Tuesday, April 29, 2014, RP Schell <rp...@schellonline.com> wrote:
> Hey everyone,
>
> We are attempting to simulate a large user load (thousands) on our
> service using JMeter 2.11. To facilitate this, we have created a JKS
> keystore containing 4000 unique PKI certificates. We have set the
> appropriate options for javax.net.ssl.keyStore and
> javax.net.ssl.keyStorePassword and added a Keystore Configuration
> component with an alias start index of 0 and alias end index of 3999.
> We verified that we can access the system as unique users using JMeter
> with this set up.
>
> We started seeing some strange results come back from the JMeter test
> runs and determined that the errors returned by our web server were
> due to the user certificate switching in the middle of a test run for
> a given JMeter thread. More specifically, we added a HTTP request
> that gives back some indication of who we are logged in as at the end
> of the test run loop. It appears every thread in JMeter is switching
> the user it is coming in as about every 5 loops of our test run. This
> occurs for each thread whether we set it to run with one or multiple
> threads.
>
> This causes problems for our testing as some HTTPS requests assume we
> are still the same user and will deny access if the user switches
> mid-way through the test run. We verified that
> https.sessioncontext.shared=false and
> https.use.cached.ssl.context=true is set in the jmeter.properties
> file. We did try other combinations of these two options w/o success.
>
> We also tried setting the implementation for HTTP requests to
> HTTPClient3.1 as well as HTTPClient4. The behavior is the same.
>
> Is this behavior intended by JMeter? The only way I can prevent the
> certificate from switching mid-test is to set the alias start and end
> index to the same number in the Keystore Configuration component. Of
> course, this is not what we want as it locks all JMeter threads into
> running as the exact same user and does not allow proper simulation of
> a multi-user load.
>
> Any insight into multi-user certificate loading in JMeter is much
> appreciated. Thanks!
>
> -Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org <javascript:;>
> For additional commands, e-mail: user-help@jmeter.apache.org<javascript:;>
>
>
--
Regards
Ubik Load Pack <http://ubikloadpack.com> Team
Follow us on Twitter <http://twitter.com/ubikloadpack>
Cordialement
L'équipe Ubik Load Pack <http://ubikloadpack.com>
Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>