You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Ar...@rbs.com on 2013/04/18 17:35:33 UTC
JDBC Sampler - Running out of memory
Hello all,
I am using JMeter 2.9 with the PerfMon. I also use Java 7.
My goal is to load test a database, so I use a JDBC Datasource and a JDBC Request.
I also use the PerfMon Metrics Collector listener, which provides me with JMX metrics.
The PerfMon Metrics requires the use of a server agent, which is a small program that runs on the server whose JMX metrics you need to collect.
The problem is that the query I am testing is returning a lot of data (millions of rows). As a result, I get a Java Heap out of memory error.
I read some guides about how to prevent running out of memory.
I don't have any other listeners. The only one is the PerfMon Metrics one, which I need to keep enabled. It works differently than the others and disabling it will not return any results.
I also increased the heap size for JMeter to 1GB, but I don't think I can increase it further (JMeter won't start).
I am suspecting that the JDBC Sampler is actually going through the resultset and doing something with the results of the query.
https://svn.apache.org/repos/asf/jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java
In my case, I would like to discard the query results. I am just interested in the JMX metrics which are collected from the PerfMon add-on (Heap memory, garbage collection times, etc.)
Am I correct in my assumption?
Am I missing something here?
Many thanks,
Argyrios Tzakas
Developer
Markets & International Banking
RBS
Bankside 3, 90-100 Southwark Street, London SE1 0SW, UK
Office: +44 2076787551
***********************************************************************************
The Royal Bank of Scotland plc. Registered in Scotland No 90312.
Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB.
Authorised and regulated by the Financial Services Authority. The
Royal Bank of Scotland N.V. is authorised and regulated by the
De Nederlandsche Bank and has its seat at Amsterdam, the
Netherlands, and is registered in the Commercial Register under
number 33002587. Registered Office: Gustav Mahlerlaan 350,
Amsterdam, The Netherlands. The Royal Bank of Scotland N.V. and
The Royal Bank of Scotland plc are authorised to act as agent for each
other in certain jurisdictions.
This e-mail message is confidential and for use by the addressee only.
If the message is received by anyone other than the addressee, please
return the message to the sender by replying to it and then delete the
message from your computer. Internet e-mails are not necessarily
secure. The Royal Bank of Scotland plc and The Royal Bank of Scotland
N.V. including its affiliates ("RBS group") does not accept responsibility
for changes made to this message after it was sent. For the protection
of RBS group and its clients and customers, and in compliance with
regulatory requirements, the contents of both incoming and outgoing
e-mail communications, which could include proprietary information and
Non-Public Personal Information, may be read by authorised persons
within RBS group other than the intended recipient(s).
Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the onward
transmission, opening or use of this message and any attachments will
not adversely affect its systems or data. No responsibility is accepted
by the RBS group in this regard and the recipient should carry out such
virus and other checks as it considers appropriate.
Visit our website at www.rbs.com
***********************************************************************************
Re: JDBC Sampler - Running out of memory
Posted by waseemfa <fa...@gmail.com>.
I am assuming your client has limited RAM because of which you are not able
to increase the heap size of JMeter. In such case why dont you try
distributing the load into different machines which will take care of the
memory problem.
-Waseem
--
View this message in context: http://jmeter.512774.n5.nabble.com/JDBC-Sampler-Running-out-of-memory-tp5716731p5716755.html
Sent from the JMeter - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
RE: JDBC Sampler - Running out of memory
Posted by Ar...@rbs.com.
Hello again,
Many thanks to those who replied to my post.
I managed to resolve the issue by building Jmeter from the source code and modifying the "AbstractJDBCTestElement.java" file.
Essentially, I modified the getStringFromResultSet function and, in particular, the while(rs.next()) and for() loops that are contained within it.
The reason I did that is because I am only interested in the JMX metrics of the JVM hosting my database engine.
My goal is to monitor things like memory usage, time spent on GC, etc. and I can just discard any data returned by the query.
The >1million rows I mentioned is typical of the queries we run here.
Again, thanks for your support and I will keep posting and contributing here as I learn more.
Argyrios Tzakas
-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 22 April 2013 12:32
To: JMeter Users List
Subject: Re: JDBC Sampler - Running out of memory
On 18 April 2013 16:35, <Ar...@rbs.com> wrote:
> Hello all,
>
> I am using JMeter 2.9 with the PerfMon. I also use Java 7.
>
> My goal is to load test a database, so I use a JDBC Datasource and a
> JDBC Request.
> I also use the PerfMon Metrics Collector listener, which provides me
> with JMX metrics.
> The PerfMon Metrics requires the use of a server agent, which is a
> small program that runs on the server whose JMX metrics you need to collect.
>
> The problem is that the query I am testing is returning a lot of data
> (millions of rows). As a result, I get a Java Heap out of memory error.
>
> I read some guides about how to prevent running out of memory.
> I don't have any other listeners. The only one is the PerfMon Metrics
> one, which I need to keep enabled. It works differently than the
> others and disabling it will not return any results.
> I also increased the heap size for JMeter to 1GB, but I don't think I
> can increase it further (JMeter won't start).
>
> I am suspecting that the JDBC Sampler is actually going through the
> resultset and doing something with the results of the query.
>
> https://svn.apache.org/repos/asf/jmeter/trunk/src/protocol/jdbc/org/ap
> ache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java
>
>
Yes, it is processing the response, because that is what you asked it to do.
Do you really need millions of rows?
Is that how the real application behaves?
What does the application do with the rows?
Maybe you could change the SQL to just count the rows, or otherwise adjust it so it returns less data.
> In my case, I would like to discard the query results. I am just
> interested in the JMX metrics which are collected from the PerfMon
> add-on (Heap memory, garbage collection times, etc.)
>
>
You could run PerfMon in a separate JMeter instance.
> Am I correct in my assumption?
> Am I missing something here?
>
>
There's a related enhancement request, but it has not been implemented yet:
https://issues.apache.org/bugzilla/show_bug.cgi?id=41921
>
> Many thanks,
>
> Argyrios Tzakas
>
***********************************************************************************
The Royal Bank of Scotland plc. Registered in Scotland No 90312.
Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB.
Authorised and regulated by the Financial Services Authority. The
Royal Bank of Scotland N.V. is authorised and regulated by the
De Nederlandsche Bank and has its seat at Amsterdam, the
Netherlands, and is registered in the Commercial Register under
number 33002587. Registered Office: Gustav Mahlerlaan 350,
Amsterdam, The Netherlands. The Royal Bank of Scotland N.V. and
The Royal Bank of Scotland plc are authorised to act as agent for each
other in certain jurisdictions.
This e-mail message is confidential and for use by the addressee only.
If the message is received by anyone other than the addressee, please
return the message to the sender by replying to it and then delete the
message from your computer. Internet e-mails are not necessarily
secure. The Royal Bank of Scotland plc and The Royal Bank of Scotland
N.V. including its affiliates ("RBS group") does not accept responsibility
for changes made to this message after it was sent. For the protection
of RBS group and its clients and customers, and in compliance with
regulatory requirements, the contents of both incoming and outgoing
e-mail communications, which could include proprietary information and
Non-Public Personal Information, may be read by authorised persons
within RBS group other than the intended recipient(s).
Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the onward
transmission, opening or use of this message and any attachments will
not adversely affect its systems or data. No responsibility is accepted
by the RBS group in this regard and the recipient should carry out such
virus and other checks as it considers appropriate.
Visit our website at www.rbs.com
***********************************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: JDBC Sampler - Running out of memory
Posted by sebb <se...@gmail.com>.
On 18 April 2013 16:35, <Ar...@rbs.com> wrote:
> Hello all,
>
> I am using JMeter 2.9 with the PerfMon. I also use Java 7.
>
> My goal is to load test a database, so I use a JDBC Datasource and a JDBC
> Request.
> I also use the PerfMon Metrics Collector listener, which provides me with
> JMX metrics.
> The PerfMon Metrics requires the use of a server agent, which is a small
> program that runs on the server whose JMX metrics you need to collect.
>
> The problem is that the query I am testing is returning a lot of data
> (millions of rows). As a result, I get a Java Heap out of memory error.
>
> I read some guides about how to prevent running out of memory.
> I don't have any other listeners. The only one is the PerfMon Metrics one,
> which I need to keep enabled. It works differently than the others and
> disabling it will not return any results.
> I also increased the heap size for JMeter to 1GB, but I don't think I can
> increase it further (JMeter won't start).
>
> I am suspecting that the JDBC Sampler is actually going through the
> resultset and doing something with the results of the query.
>
> https://svn.apache.org/repos/asf/jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java
>
>
Yes, it is processing the response, because that is what you asked it to do.
Do you really need millions of rows?
Is that how the real application behaves?
What does the application do with the rows?
Maybe you could change the SQL to just count the rows, or otherwise adjust
it so it returns less data.
> In my case, I would like to discard the query results. I am just
> interested in the JMX metrics which are collected from the PerfMon add-on
> (Heap memory, garbage collection times, etc.)
>
>
You could run PerfMon in a separate JMeter instance.
> Am I correct in my assumption?
> Am I missing something here?
>
>
There's a related enhancement request, but it has not been implemented yet:
https://issues.apache.org/bugzilla/show_bug.cgi?id=41921
>
> Many thanks,
>
> Argyrios Tzakas
> Developer
> Markets & International Banking
> RBS
> Bankside 3, 90-100 Southwark Street, London SE1 0SW, UK
> Office: +44 2076787551
>
>
>
> ***********************************************************************************
> The Royal Bank of Scotland plc. Registered in Scotland No 90312.
> Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB.
> Authorised and regulated by the Financial Services Authority. The
> Royal Bank of Scotland N.V. is authorised and regulated by the
> De Nederlandsche Bank and has its seat at Amsterdam, the
> Netherlands, and is registered in the Commercial Register under
> number 33002587. Registered Office: Gustav Mahlerlaan 350,
> Amsterdam, The Netherlands. The Royal Bank of Scotland N.V. and
> The Royal Bank of Scotland plc are authorised to act as agent for each
> other in certain jurisdictions.
>
> This e-mail message is confidential and for use by the addressee only.
> If the message is received by anyone other than the addressee, please
> return the message to the sender by replying to it and then delete the
> message from your computer. Internet e-mails are not necessarily
> secure. The Royal Bank of Scotland plc and The Royal Bank of Scotland
> N.V. including its affiliates ("RBS group") does not accept responsibility
> for changes made to this message after it was sent. For the protection
> of RBS group and its clients and customers, and in compliance with
> regulatory requirements, the contents of both incoming and outgoing
> e-mail communications, which could include proprietary information and
> Non-Public Personal Information, may be read by authorised persons
> within RBS group other than the intended recipient(s).
>
> Whilst all reasonable care has been taken to avoid the transmission of
> viruses, it is the responsibility of the recipient to ensure that the
> onward
> transmission, opening or use of this message and any attachments will
> not adversely affect its systems or data. No responsibility is accepted
> by the RBS group in this regard and the recipient should carry out such
> virus and other checks as it considers appropriate.
>
> Visit our website at www.rbs.com
>
>
> ***********************************************************************************
>