You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Lynch Kieran <k....@reply.com> on 2019/11/05 20:02:14 UTC
Jmeter System Preferences Lock Error
Hi guys,
Potentially a strange one for you... I'm running a test on an EC2 instance with 64G available memory and jmeter's heap sized appropriately (currently Xmx40g) configured for 6K concurrent users and 200TPS. When the test reaches 4k users I get the following error message output:
WARNING: Could not lock User prefs. Unix error code 24.
Nov 05, 2019 7:49:00 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
And in my jmeter.log a stream of these:
2019-11-05 19:47:44,512 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
2019-11-05 19:47:44,612 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
2019-11-05 19:47:44,612 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
2019-11-05 19:47:44,612 ERROR o.a.j.c.CSVDataSet: java.io.IOException: Stream closed
Test errors are out of control from this point onwards (60 - 70%) despite application health looking good - I'm assuming because jmeter is unable to send CSV parameterised requests?
I did a little digging and it seems to be a Java install specific issue? Although I find it strange that I get these errors at 4K threads every time. I am running jmeter as root and my Java system preferences file is writable for all:
lrwxrwxrwx 1 root root 35 Nov 5 18:15 .systemPrefs -> /usr/java/latest/.java/.systemPrefs
java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
echo $JAVA_HOME
/usr/java/jre1.8.0_231-amd64
Any help or info from someone who has experienced this before greatly appreciated.
Kind Regards,
Kieran
Kieran Lynch
Portaltech Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.lynch@reply.com<ma...@reply.com>
www.reply.com
[Portaltech Reply]
Re: Jmeter System Preferences Lock Error
Posted by Lynch Kieran <k....@reply.com>.
Hi glinius,
Appreciate the detailed response and links.
You were correct with your second hypothesis, turns out there was a hard limit of 4096 for file descriptors on the EC2 instance. I edited /etc/security/limits.conf and added the following two lines:
root soft nofile 6114
root hard nofile 12288
and that appears to have resolved the issue.
Thanks again for saving me!
Kieran
Kieran Lynch
Portaltech Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.lynch@reply.com<ma...@reply.com>
www.reply.com
[Portaltech Reply]
________________________________
From: glinius@live.com <gl...@live.com>
Sent: 07 November 2019 09:38
To: jmeter-user@jakarta.apache.org <jm...@jakarta.apache.org>
Subject: Re: Jmeter System Preferences Lock Error
Looking into JMeter Bug 61279
<https://bz.apache.org/bugzilla/show_bug.cgi?id=61279> it isn't something
connected with JMeter, I can think of the following reasons for the
behaviour:1. You have more than 1 JVM
<https://en.wikipedia.org/wiki/Java_virtual_machine> running on the EC2
machine and these JVMs are suffering from a form of a race condition
<https://en.wikipedia.org/wiki/Race_condition> when trying to obtain the
lock on the same file to store the preferences2. There is a problem with
Linux file permissions and JVM, i.e. you're running out of free handles
<https://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html> The
solution for point 1 would be explicitly specifying the store for
systemRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#systemRoot()>
and userRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#userRoot()>
for the JVM which hosts JMeter so it would use the separate preferences
storage not shared with other JVMs. The properties can be defined either via
-D command-line argument
<https://jmeter.apache.org/usermanual/get-started.html#override> like:
> ./jmeter -Djava.util.prefs.systemRoot=/some/folder/.jmeter
> -Djava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs
In order to make the changes persistent you can add the same lines to
/system.properties/ file (lives in "bin" folder of your JMeter installation)
> java.util.prefs.systemRoot=/some/folder/.jmeterjava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs
More information: Apache JMeter Properties Customization Guide
<https://www.blazemeter.com/blog/apache-jmeter-properties-customization/>
In case of point 2 I would recommend checking / raising Linux process
limits <https://gerardnico.com/os/linux/limits.conf>
--
Sent from: http://www.jmeter-archive.org/JMeter-User-f512775.html
Re: Jmeter System Preferences Lock Error
Posted by "glinius@live.com" <gl...@live.com>.
Looking into JMeter Bug 61279
<https://bz.apache.org/bugzilla/show_bug.cgi?id=61279> it isn't something
connected with JMeter, I can think of the following reasons for the
behaviour:1. You have more than 1 JVM
<https://en.wikipedia.org/wiki/Java_virtual_machine> running on the EC2
machine and these JVMs are suffering from a form of a race condition
<https://en.wikipedia.org/wiki/Race_condition> when trying to obtain the
lock on the same file to store the preferences2. There is a problem with
Linux file permissions and JVM, i.e. you're running out of free handles
<https://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html> The
solution for point 1 would be explicitly specifying the store for
systemRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#systemRoot()>
and userRoot
<https://docs.oracle.com/javase/7/docs/api/java/util/prefs/PreferencesFactory.html#userRoot()>
for the JVM which hosts JMeter so it would use the separate preferences
storage not shared with other JVMs. The properties can be defined either via
-D command-line argument
<https://jmeter.apache.org/usermanual/get-started.html#override> like:
> ./jmeter -Djava.util.prefs.systemRoot=/some/folder/.jmeter
> -Djava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs
In order to make the changes persistent you can add the same lines to
/system.properties/ file (lives in "bin" folder of your JMeter installation)
> java.util.prefs.systemRoot=/some/folder/.jmeterjava.util.prefs.userRoot=/some/folder/.jmeter/.userPrefs
More information: Apache JMeter Properties Customization Guide
<https://www.blazemeter.com/blog/apache-jmeter-properties-customization/>
In case of point 2 I would recommend checking / raising Linux process
limits <https://gerardnico.com/os/linux/limits.conf>
--
Sent from: http://www.jmeter-archive.org/JMeter-User-f512775.html