You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Carsten Petersen <cp...@testhuset.dk> on 2012/06/07 15:04:28 UTC
Saving large (>500Mb) http response to a file.
Hi.
I am trying to verify a large (>500Mb) response from a HTTP request, but whenever I run my TestPlan
the thread fails after running out of memory.
I have added a "Save Responses to a file" Listener, but it looks like it is the HTTP Request sampler that
fails when handling all the data in memory after the reception completes - see error ourput below.
Console window output:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1912.hprof ...
Heap dump file created [558234258 bytes in 6.291 secs]
JMeter log file output:
2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
at java.lang.Thread.run(Unknown Source)
I have already increased the heap allocation to its max on my hardware (-Xmx1G), but without success. I guess I could
add memory, but since I have to run multiple downloads at the same time, it does not look realistic.
I can enable the "Save response as MD5 hash" option and have my TestPlan succeed, but then the output file from the
run only contain the MD5 hash and since my response data is somewhat dynamic it is of no real use.
I can see that JMeter allows a large file to be uploaded by reading it as needed during transmission, so I guess I need
the reverse of this i.e. store the response to a file while receiving it without keeping it in memory. I guess something
like having the "Save response as MD5 hash" option dump the data to a file after its calculation are done.
Does anybody know how I can do this ?
Thanks in advance,
Carsten.
P.S. I am using the following JMeter:
2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM) Client VM
2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
SV: Saving large (>500Mb) http response to a file.
Posted by Carsten Petersen <cp...@testhuset.dk>.
Hi Deepak.
Currently I am only running one user, but I need to run at least 100 and even more
in trying to stress the system entirely.
This also means that tuning does not look like the right way to go. If I download
500Mb times 100 users that could translate into 50Gb of storage and that is even
before JMeter starts with the copy that fails for me.
This is why I need direct storage of the response data.
Thanks, Carsten.
-----Oprindelig meddelelse-----
Fra: Deepak Goel [mailto:deicool@gmail.com]
Sendt: to 07-06-2012 17:15
Til: JMeter Users List
Emne: Re: Saving large (>500Mb) http response to a file.
Hey
How many users are you running? Is it just one? You might have to
refine your other java heap settings too
:)
Deepak
On 6/7/12, Cyril SANTUNE <cy...@gmail.com> wrote:
> Hi,
>
> I only do test with big files (10MB, 100MB, ...) and I have same trouble.
> So I use beanshell script and curl do make the transfert.
> But now, with the version 2.7, you can use external command with the
> sampler "OS Process Sampler".
>
> Cyril
>
>
>
> On Thu, Jun 7, 2012 at 9:04 AM, Carsten Petersen <cp...@testhuset.dk> wrote:
>
>> Hi.
>>
>> I am trying to verify a large (>500Mb) response from a HTTP request, but
>> whenever I run my TestPlan
>> the thread fails after running out of memory.
>>
>> I have added a "Save Responses to a file" Listener, but it looks like it
>> is the HTTP Request sampler that
>> fails when handling all the data in memory after the reception completes
>> -
>> see error ourput below.
>>
>> Console window output:
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid1912.hprof ...
>> Heap dump file created [558234258 bytes in 6.291 secs]
>>
>> JMeter log file output:
>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed!
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Unknown Source)
>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>> at
>> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>> at
>> org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>> at java.lang.Thread.run(Unknown Source)
>>
>> I have already increased the heap allocation to its max on my hardware
>> (-Xmx1G), but without success. I guess I could
>> add memory, but since I have to run multiple downloads at the same time,
>> it does not look realistic.
>>
>> I can enable the "Save response as MD5 hash" option and have my TestPlan
>> succeed, but then the output file from the
>> run only contain the MD5 hash and since my response data is somewhat
>> dynamic it is of no real use.
>>
>> I can see that JMeter allows a large file to be uploaded by reading it as
>> needed during transmission, so I guess I need
>> the reverse of this i.e. store the response to a file while receiving it
>> without keeping it in memory. I guess something
>> like having the "Save response as MD5 hash" option dump the data to a
>> file
>> after its calculation are done.
>>
>> Does anybody know how I can do this ?
>>
>> Thanks in advance,
>> Carsten.
>>
>> P.S. I am using the following JMeter:
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM)
>> Client VM
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>>
>
--
Namaskara~Nalama~Guten Tag~Bonjour
--
Keigu
Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Saving large (>500Mb) http response to a file.
Posted by Deepak Goel <de...@gmail.com>.
Hey
How many users are you running? Is it just one? You might have to
refine your other java heap settings too
:)
Deepak
On 6/7/12, Cyril SANTUNE <cy...@gmail.com> wrote:
> Hi,
>
> I only do test with big files (10MB, 100MB, ...) and I have same trouble.
> So I use beanshell script and curl do make the transfert.
> But now, with the version 2.7, you can use external command with the
> sampler "OS Process Sampler".
>
> Cyril
>
>
>
> On Thu, Jun 7, 2012 at 9:04 AM, Carsten Petersen <cp...@testhuset.dk> wrote:
>
>> Hi.
>>
>> I am trying to verify a large (>500Mb) response from a HTTP request, but
>> whenever I run my TestPlan
>> the thread fails after running out of memory.
>>
>> I have added a "Save Responses to a file" Listener, but it looks like it
>> is the HTTP Request sampler that
>> fails when handling all the data in memory after the reception completes
>> -
>> see error ourput below.
>>
>> Console window output:
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid1912.hprof ...
>> Heap dump file created [558234258 bytes in 6.291 secs]
>>
>> JMeter log file output:
>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed!
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Unknown Source)
>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>> at
>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>> at
>> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>> at
>> org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>> at java.lang.Thread.run(Unknown Source)
>>
>> I have already increased the heap allocation to its max on my hardware
>> (-Xmx1G), but without success. I guess I could
>> add memory, but since I have to run multiple downloads at the same time,
>> it does not look realistic.
>>
>> I can enable the "Save response as MD5 hash" option and have my TestPlan
>> succeed, but then the output file from the
>> run only contain the MD5 hash and since my response data is somewhat
>> dynamic it is of no real use.
>>
>> I can see that JMeter allows a large file to be uploaded by reading it as
>> needed during transmission, so I guess I need
>> the reverse of this i.e. store the response to a file while receiving it
>> without keeping it in memory. I guess something
>> like having the "Save response as MD5 hash" option dump the data to a
>> file
>> after its calculation are done.
>>
>> Does anybody know how I can do this ?
>>
>> Thanks in advance,
>> Carsten.
>>
>> P.S. I am using the following JMeter:
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM)
>> Client VM
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>>
>
--
Namaskara~Nalama~Guten Tag~Bonjour
--
Keigu
Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Saving large (>500Mb) http response to a file.
Posted by Cyril SANTUNE <cy...@gmail.com>.
Hi,
I only do test with big files (10MB, 100MB, ...) and I have same trouble.
So I use beanshell script and curl do make the transfert.
But now, with the version 2.7, you can use external command with the
sampler "OS Process Sampler".
Cyril
On Thu, Jun 7, 2012 at 9:04 AM, Carsten Petersen <cp...@testhuset.dk> wrote:
> Hi.
>
> I am trying to verify a large (>500Mb) response from a HTTP request, but
> whenever I run my TestPlan
> the thread fails after running out of memory.
>
> I have added a "Save Responses to a file" Listener, but it looks like it
> is the HTTP Request sampler that
> fails when handling all the data in memory after the reception completes -
> see error ourput below.
>
> Console window output:
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid1912.hprof ...
> Heap dump file created [558234258 bytes in 6.291 secs]
>
> JMeter log file output:
> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed!
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
> at
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
> at
> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
> at java.lang.Thread.run(Unknown Source)
>
> I have already increased the heap allocation to its max on my hardware
> (-Xmx1G), but without success. I guess I could
> add memory, but since I have to run multiple downloads at the same time,
> it does not look realistic.
>
> I can enable the "Save response as MD5 hash" option and have my TestPlan
> succeed, but then the output file from the
> run only contain the MD5 hash and since my response data is somewhat
> dynamic it is of no real use.
>
> I can see that JMeter allows a large file to be uploaded by reading it as
> needed during transmission, so I guess I need
> the reverse of this i.e. store the response to a file while receiving it
> without keeping it in memory. I guess something
> like having the "Save response as MD5 hash" option dump the data to a file
> after its calculation are done.
>
> Does anybody know how I can do this ?
>
> Thanks in advance,
> Carsten.
>
> P.S. I am using the following JMeter:
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM)
> Client VM
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>
Re: Saving large (>500Mb) http response to a file.
Posted by sebb <se...@gmail.com>.
On 8 June 2012 10:33, Carsten Petersen <cp...@testhuset.dk> wrote:
> I was looking at running an external program (wget or curl) to do this, but I need to extract
> some information from the HTTP header and I have not yet seen how I can return some of it.
>
> The OS Process Sampler was a good idea (I missed that one) because I can at least match the return
> code from the external process, but I still need the header as well.
The BeanShell sampler would allow you to do it all.
> I was thinking about making my own "web server" from Python or Ruby to call and have it do the'
> HTTP get, because then I could return the header as a response to the HTTP Request Sampler in JMeter.
Or perhaps you could make two requests: HEAD and GET
> I can see in the code (HTTPSamplerBase.java) that the readResponse function does store all of the
> received data if MD5 is not used and that it drops the data if MD5 is used. Since I am a complete
> newbie in the JMeter code I do not yet know the structure, but if I could get the file name from a
> child Save Response to a file Listener it would be easy to add a few lines to have readResponse
> write to received data to a file as well. I will see if I find the time to look at this.
>
> Thank you for your help so far.
> Carsten.
>
>
> -----Oprindelig meddelelse-----
> Fra: sebb [mailto:sebbaz@gmail.com]
> Sendt: fr 08-06-2012 10:32
> Til: JMeter Users List
> Emne: Re: Saving large (>500Mb) http response to a file.
>
> On 8 June 2012 08:12, Carsten Petersen <cp...@testhuset.dk> wrote:
>> Hi Sebb.
>>
>> That was what I was afraid of.
>>
>> I have not looked at the JMeter code yet, but it looks like the sampler stores the entire
>> response internally before making a copy for the listener that could store the data.
>
> Yes.
>
>> For this to work I guess the sampler must store the data itself, but that could hurt the
>> modularity of JMeter.
>
> Yes, the sampler must store the data directly.
>
> Won't necessarily affect JMeter modularity.
>
>> Because of the time I have left, I might not have the time dig into this in details through.
>
> As already mentioned, a workround would be to use the OS Process
> Sampler with curl or wget.
>
> Or you could use a BeanShell Sampler to load and store the file.
>
>> Thank you,
>> Carsten.
>>
>>
>> -----Oprindelig meddelelse-----
>> Fra: sebb [mailto:sebbaz@gmail.com]
>> Sendt: to 07-06-2012 17:41
>> Til: JMeter Users List
>> Emne: Re: Saving large (>500Mb) http response to a file.
>>
>> On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
>>> Hi.
>>>
>>> I am trying to verify a large (>500Mb) response from a HTTP request, but whenever I run my TestPlan
>>> the thread fails after running out of memory.
>>>
>>> I have added a "Save Responses to a file" Listener, but it looks like it is the HTTP Request sampler that
>>> fails when handling all the data in memory after the reception completes - see error ourput below.
>>>
>>> Console window output:
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid1912.hprof ...
>>> Heap dump file created [558234258 bytes in 6.291 secs]
>>>
>>> JMeter log file output:
>>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space
>>> at java.util.Arrays.copyOf(Unknown Source)
>>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>>> at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>>> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>> I have already increased the heap allocation to its max on my hardware (-Xmx1G), but without success. I guess I could
>>> add memory, but since I have to run multiple downloads at the same time, it does not look realistic.
>>>
>>> I can enable the "Save response as MD5 hash" option and have my TestPlan succeed, but then the output file from the
>>> run only contain the MD5 hash and since my response data is somewhat dynamic it is of no real use.
>>>
>>> I can see that JMeter allows a large file to be uploaded by reading it as needed during transmission, so I guess I need
>>> the reverse of this i.e. store the response to a file while receiving it without keeping it in memory. I guess something
>>> like having the "Save response as MD5 hash" option dump the data to a file after its calculation are done.
>>>
>>> Does anybody know how I can do this ?
>>
>> I think this will need a change to the JMeter HttpSampler code.
>> Listeners can only save what the sampler stores in the sample result.
>>
>> I suggest you file an enhancement request in Bugzilla.
>>
>>> Thanks in advance,
>>> Carsten.
>>>
>>> P.S. I am using the following JMeter:
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM) Client VM
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
SV: Saving large (>500Mb) http response to a file.
Posted by Carsten Petersen <cp...@testhuset.dk>.
I was looking at running an external program (wget or curl) to do this, but I need to extract
some information from the HTTP header and I have not yet seen how I can return some of it.
The OS Process Sampler was a good idea (I missed that one) because I can at least match the return
code from the external process, but I still need the header as well.
I was thinking about making my own "web server" from Python or Ruby to call and have it do the'
HTTP get, because then I could return the header as a response to the HTTP Request Sampler in JMeter.
I can see in the code (HTTPSamplerBase.java) that the readResponse function does store all of the
received data if MD5 is not used and that it drops the data if MD5 is used. Since I am a complete
newbie in the JMeter code I do not yet know the structure, but if I could get the file name from a
child Save Response to a file Listener it would be easy to add a few lines to have readResponse
write to received data to a file as well. I will see if I find the time to look at this.
Thank you for your help so far.
Carsten.
-----Oprindelig meddelelse-----
Fra: sebb [mailto:sebbaz@gmail.com]
Sendt: fr 08-06-2012 10:32
Til: JMeter Users List
Emne: Re: Saving large (>500Mb) http response to a file.
On 8 June 2012 08:12, Carsten Petersen <cp...@testhuset.dk> wrote:
> Hi Sebb.
>
> That was what I was afraid of.
>
> I have not looked at the JMeter code yet, but it looks like the sampler stores the entire
> response internally before making a copy for the listener that could store the data.
Yes.
> For this to work I guess the sampler must store the data itself, but that could hurt the
> modularity of JMeter.
Yes, the sampler must store the data directly.
Won't necessarily affect JMeter modularity.
> Because of the time I have left, I might not have the time dig into this in details through.
As already mentioned, a workround would be to use the OS Process
Sampler with curl or wget.
Or you could use a BeanShell Sampler to load and store the file.
> Thank you,
> Carsten.
>
>
> -----Oprindelig meddelelse-----
> Fra: sebb [mailto:sebbaz@gmail.com]
> Sendt: to 07-06-2012 17:41
> Til: JMeter Users List
> Emne: Re: Saving large (>500Mb) http response to a file.
>
> On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
>> Hi.
>>
>> I am trying to verify a large (>500Mb) response from a HTTP request, but whenever I run my TestPlan
>> the thread fails after running out of memory.
>>
>> I have added a "Save Responses to a file" Listener, but it looks like it is the HTTP Request sampler that
>> fails when handling all the data in memory after the reception completes - see error ourput below.
>>
>> Console window output:
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid1912.hprof ...
>> Heap dump file created [558234258 bytes in 6.291 secs]
>>
>> JMeter log file output:
>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Unknown Source)
>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>> at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>> at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>> at java.lang.Thread.run(Unknown Source)
>>
>> I have already increased the heap allocation to its max on my hardware (-Xmx1G), but without success. I guess I could
>> add memory, but since I have to run multiple downloads at the same time, it does not look realistic.
>>
>> I can enable the "Save response as MD5 hash" option and have my TestPlan succeed, but then the output file from the
>> run only contain the MD5 hash and since my response data is somewhat dynamic it is of no real use.
>>
>> I can see that JMeter allows a large file to be uploaded by reading it as needed during transmission, so I guess I need
>> the reverse of this i.e. store the response to a file while receiving it without keeping it in memory. I guess something
>> like having the "Save response as MD5 hash" option dump the data to a file after its calculation are done.
>>
>> Does anybody know how I can do this ?
>
> I think this will need a change to the JMeter HttpSampler code.
> Listeners can only save what the sampler stores in the sample result.
>
> I suggest you file an enhancement request in Bugzilla.
>
>> Thanks in advance,
>> Carsten.
>>
>> P.S. I am using the following JMeter:
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM) Client VM
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
SV: Saving large (>500Mb) http response to a file.
Posted by Carsten Petersen <cp...@testhuset.dk>.
I am sorry, but I am not sure I can follow you here.
I know that the ramp-up will make sure that I do not download for all 100 users
at the same time, but I will still have a huge memory problem.
Thanks,
Carsten.
-----Oprindelig meddelelse-----
Fra: Deepak Goel [mailto:deicool@gmail.com]
Sendt: fr 08-06-2012 11:10
Til: JMeter Users List
Emne: Re: Saving large (>500Mb) http response to a file.
Hey
It might be a good idea to tune your code and environment (system)
perfectly for 1 user. And then move on to 100 users. Say if you go
from 1 user to 100users, and your mistake for 1 user is around 25%
that increases exponentially and not serially to 80-90%.
:)
Deepak
On 6/8/12, sebb <se...@gmail.com> wrote:
> On 8 June 2012 08:12, Carsten Petersen <cp...@testhuset.dk> wrote:
>> Hi Sebb.
>>
>> That was what I was afraid of.
>>
>> I have not looked at the JMeter code yet, but it looks like the sampler
>> stores the entire
>> response internally before making a copy for the listener that could store
>> the data.
>
> Yes.
>
>> For this to work I guess the sampler must store the data itself, but that
>> could hurt the
>> modularity of JMeter.
>
> Yes, the sampler must store the data directly.
>
> Won't necessarily affect JMeter modularity.
>
>> Because of the time I have left, I might not have the time dig into this
>> in details through.
>
> As already mentioned, a workround would be to use the OS Process
> Sampler with curl or wget.
>
> Or you could use a BeanShell Sampler to load and store the file.
>
>> Thank you,
>> Carsten.
>>
>>
>> -----Oprindelig meddelelse-----
>> Fra: sebb [mailto:sebbaz@gmail.com]
>> Sendt: to 07-06-2012 17:41
>> Til: JMeter Users List
>> Emne: Re: Saving large (>500Mb) http response to a file.
>>
>> On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
>>> Hi.
>>>
>>> I am trying to verify a large (>500Mb) response from a HTTP request, but
>>> whenever I run my TestPlan
>>> the thread fails after running out of memory.
>>>
>>> I have added a "Save Responses to a file" Listener, but it looks like it
>>> is the HTTP Request sampler that
>>> fails when handling all the data in memory after the reception completes
>>> - see error ourput below.
>>>
>>> Console window output:
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid1912.hprof ...
>>> Heap dump file created [558234258 bytes in 6.291 secs]
>>>
>>> JMeter log file output:
>>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed!
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.util.Arrays.copyOf(Unknown Source)
>>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>>> at
>>> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>>> at
>>> org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>> I have already increased the heap allocation to its max on my hardware
>>> (-Xmx1G), but without success. I guess I could
>>> add memory, but since I have to run multiple downloads at the same time,
>>> it does not look realistic.
>>>
>>> I can enable the "Save response as MD5 hash" option and have my TestPlan
>>> succeed, but then the output file from the
>>> run only contain the MD5 hash and since my response data is somewhat
>>> dynamic it is of no real use.
>>>
>>> I can see that JMeter allows a large file to be uploaded by reading it as
>>> needed during transmission, so I guess I need
>>> the reverse of this i.e. store the response to a file while receiving it
>>> without keeping it in memory. I guess something
>>> like having the "Save response as MD5 hash" option dump the data to a
>>> file after its calculation are done.
>>>
>>> Does anybody know how I can do this ?
>>
>> I think this will need a change to the JMeter HttpSampler code.
>> Listeners can only save what the sampler stores in the sample result.
>>
>> I suggest you file an enhancement request in Bugzilla.
>>
>>> Thanks in advance,
>>> Carsten.
>>>
>>> P.S. I am using the following JMeter:
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM)
>>> Client VM
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
--
Namaskara~Nalama~Guten Tag~Bonjour
--
Keigu
Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Saving large (>500Mb) http response to a file.
Posted by Deepak Goel <de...@gmail.com>.
Hey
It might be a good idea to tune your code and environment (system)
perfectly for 1 user. And then move on to 100 users. Say if you go
from 1 user to 100users, and your mistake for 1 user is around 25%
that increases exponentially and not serially to 80-90%.
:)
Deepak
On 6/8/12, sebb <se...@gmail.com> wrote:
> On 8 June 2012 08:12, Carsten Petersen <cp...@testhuset.dk> wrote:
>> Hi Sebb.
>>
>> That was what I was afraid of.
>>
>> I have not looked at the JMeter code yet, but it looks like the sampler
>> stores the entire
>> response internally before making a copy for the listener that could store
>> the data.
>
> Yes.
>
>> For this to work I guess the sampler must store the data itself, but that
>> could hurt the
>> modularity of JMeter.
>
> Yes, the sampler must store the data directly.
>
> Won't necessarily affect JMeter modularity.
>
>> Because of the time I have left, I might not have the time dig into this
>> in details through.
>
> As already mentioned, a workround would be to use the OS Process
> Sampler with curl or wget.
>
> Or you could use a BeanShell Sampler to load and store the file.
>
>> Thank you,
>> Carsten.
>>
>>
>> -----Oprindelig meddelelse-----
>> Fra: sebb [mailto:sebbaz@gmail.com]
>> Sendt: to 07-06-2012 17:41
>> Til: JMeter Users List
>> Emne: Re: Saving large (>500Mb) http response to a file.
>>
>> On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
>>> Hi.
>>>
>>> I am trying to verify a large (>500Mb) response from a HTTP request, but
>>> whenever I run my TestPlan
>>> the thread fails after running out of memory.
>>>
>>> I have added a "Save Responses to a file" Listener, but it looks like it
>>> is the HTTP Request sampler that
>>> fails when handling all the data in memory after the reception completes
>>> - see error ourput below.
>>>
>>> Console window output:
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid1912.hprof ...
>>> Heap dump file created [558234258 bytes in 6.291 secs]
>>>
>>> JMeter log file output:
>>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed!
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.util.Arrays.copyOf(Unknown Source)
>>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>>> at
>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>>> at
>>> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>>> at
>>> org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>> I have already increased the heap allocation to its max on my hardware
>>> (-Xmx1G), but without success. I guess I could
>>> add memory, but since I have to run multiple downloads at the same time,
>>> it does not look realistic.
>>>
>>> I can enable the "Save response as MD5 hash" option and have my TestPlan
>>> succeed, but then the output file from the
>>> run only contain the MD5 hash and since my response data is somewhat
>>> dynamic it is of no real use.
>>>
>>> I can see that JMeter allows a large file to be uploaded by reading it as
>>> needed during transmission, so I guess I need
>>> the reverse of this i.e. store the response to a file while receiving it
>>> without keeping it in memory. I guess something
>>> like having the "Save response as MD5 hash" option dump the data to a
>>> file after its calculation are done.
>>>
>>> Does anybody know how I can do this ?
>>
>> I think this will need a change to the JMeter HttpSampler code.
>> Listeners can only save what the sampler stores in the sample result.
>>
>> I suggest you file an enhancement request in Bugzilla.
>>
>>> Thanks in advance,
>>> Carsten.
>>>
>>> P.S. I am using the following JMeter:
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM)
>>> Client VM
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
--
Namaskara~Nalama~Guten Tag~Bonjour
--
Keigu
Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Saving large (>500Mb) http response to a file.
Posted by sebb <se...@gmail.com>.
On 8 June 2012 08:12, Carsten Petersen <cp...@testhuset.dk> wrote:
> Hi Sebb.
>
> That was what I was afraid of.
>
> I have not looked at the JMeter code yet, but it looks like the sampler stores the entire
> response internally before making a copy for the listener that could store the data.
Yes.
> For this to work I guess the sampler must store the data itself, but that could hurt the
> modularity of JMeter.
Yes, the sampler must store the data directly.
Won't necessarily affect JMeter modularity.
> Because of the time I have left, I might not have the time dig into this in details through.
As already mentioned, a workround would be to use the OS Process
Sampler with curl or wget.
Or you could use a BeanShell Sampler to load and store the file.
> Thank you,
> Carsten.
>
>
> -----Oprindelig meddelelse-----
> Fra: sebb [mailto:sebbaz@gmail.com]
> Sendt: to 07-06-2012 17:41
> Til: JMeter Users List
> Emne: Re: Saving large (>500Mb) http response to a file.
>
> On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
>> Hi.
>>
>> I am trying to verify a large (>500Mb) response from a HTTP request, but whenever I run my TestPlan
>> the thread fails after running out of memory.
>>
>> I have added a "Save Responses to a file" Listener, but it looks like it is the HTTP Request sampler that
>> fails when handling all the data in memory after the reception completes - see error ourput below.
>>
>> Console window output:
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid1912.hprof ...
>> Heap dump file created [558234258 bytes in 6.291 secs]
>>
>> JMeter log file output:
>> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Unknown Source)
>> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
>> at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
>> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
>> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
>> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
>> at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
>> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
>> at java.lang.Thread.run(Unknown Source)
>>
>> I have already increased the heap allocation to its max on my hardware (-Xmx1G), but without success. I guess I could
>> add memory, but since I have to run multiple downloads at the same time, it does not look realistic.
>>
>> I can enable the "Save response as MD5 hash" option and have my TestPlan succeed, but then the output file from the
>> run only contain the MD5 hash and since my response data is somewhat dynamic it is of no real use.
>>
>> I can see that JMeter allows a large file to be uploaded by reading it as needed during transmission, so I guess I need
>> the reverse of this i.e. store the response to a file while receiving it without keeping it in memory. I guess something
>> like having the "Save response as MD5 hash" option dump the data to a file after its calculation are done.
>>
>> Does anybody know how I can do this ?
>
> I think this will need a change to the JMeter HttpSampler code.
> Listeners can only save what the sampler stores in the sample result.
>
> I suggest you file an enhancement request in Bugzilla.
>
>> Thanks in advance,
>> Carsten.
>>
>> P.S. I am using the following JMeter:
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM) Client VM
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
>> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
SV: Saving large (>500Mb) http response to a file.
Posted by Carsten Petersen <cp...@testhuset.dk>.
Hi Sebb.
That was what I was afraid of.
I have not looked at the JMeter code yet, but it looks like the sampler stores the entire
response internally before making a copy for the listener that could store the data.
For this to work I guess the sampler must store the data itself, but that could hurt the
modularity of JMeter.
Because of the time I have left, I might not have the time dig into this in details through.
Thank you,
Carsten.
-----Oprindelig meddelelse-----
Fra: sebb [mailto:sebbaz@gmail.com]
Sendt: to 07-06-2012 17:41
Til: JMeter Users List
Emne: Re: Saving large (>500Mb) http response to a file.
On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
> Hi.
>
> I am trying to verify a large (>500Mb) response from a HTTP request, but whenever I run my TestPlan
> the thread fails after running out of memory.
>
> I have added a "Save Responses to a file" Listener, but it looks like it is the HTTP Request sampler that
> fails when handling all the data in memory after the reception completes - see error ourput below.
>
> Console window output:
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid1912.hprof ...
> Heap dump file created [558234258 bytes in 6.291 secs]
>
> JMeter log file output:
> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
> at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
> at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
> at java.lang.Thread.run(Unknown Source)
>
> I have already increased the heap allocation to its max on my hardware (-Xmx1G), but without success. I guess I could
> add memory, but since I have to run multiple downloads at the same time, it does not look realistic.
>
> I can enable the "Save response as MD5 hash" option and have my TestPlan succeed, but then the output file from the
> run only contain the MD5 hash and since my response data is somewhat dynamic it is of no real use.
>
> I can see that JMeter allows a large file to be uploaded by reading it as needed during transmission, so I guess I need
> the reverse of this i.e. store the response to a file while receiving it without keeping it in memory. I guess something
> like having the "Save response as MD5 hash" option dump the data to a file after its calculation are done.
>
> Does anybody know how I can do this ?
I think this will need a change to the JMeter HttpSampler code.
Listeners can only save what the sampler stores in the sample result.
I suggest you file an enhancement request in Bugzilla.
> Thanks in advance,
> Carsten.
>
> P.S. I am using the following JMeter:
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM) Client VM
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Saving large (>500Mb) http response to a file.
Posted by sebb <se...@gmail.com>.
On 7 June 2012 14:04, Carsten Petersen <cp...@testhuset.dk> wrote:
> Hi.
>
> I am trying to verify a large (>500Mb) response from a HTTP request, but whenever I run my TestPlan
> the thread fails after running out of memory.
>
> I have added a "Save Responses to a file" Listener, but it looks like it is the HTTP Request sampler that
> fails when handling all the data in memory after the reception completes - see error ourput below.
>
> Console window output:
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid1912.hprof ...
> Heap dump file created [558234258 bytes in 6.291 secs]
>
> JMeter log file output:
> 2012/06/07 14:51:53 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.io.ByteArrayOutputStream.toByteArray(Unknown Source)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1635)
> at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:236)
> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.readResponse(HTTPJavaImpl.java:288)
> at org.apache.jmeter.protocol.http.sampler.HTTPJavaImpl.sample(HTTPJavaImpl.java:518)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1060)
> at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1049)
> at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442)
> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
> at java.lang.Thread.run(Unknown Source)
>
> I have already increased the heap allocation to its max on my hardware (-Xmx1G), but without success. I guess I could
> add memory, but since I have to run multiple downloads at the same time, it does not look realistic.
>
> I can enable the "Save response as MD5 hash" option and have my TestPlan succeed, but then the output file from the
> run only contain the MD5 hash and since my response data is somewhat dynamic it is of no real use.
>
> I can see that JMeter allows a large file to be uploaded by reading it as needed during transmission, so I guess I need
> the reverse of this i.e. store the response to a file while receiving it without keeping it in memory. I guess something
> like having the "Save response as MD5 hash" option dump the data to a file after its calculation are done.
>
> Does anybody know how I can do this ?
I think this will need a change to the JMeter HttpSampler code.
Listeners can only save what the sampler stores in the sample result.
I suggest you file an enhancement request in Bugzilla.
> Thanks in advance,
> Carsten.
>
> P.S. I am using the following JMeter:
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: Version 2.7 r1342410
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.version=1.7.0_04
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: java.vm.name=Java HotSpot(TM) Client VM
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.name=Windows 7
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.arch=x86
> 2012/06/07 14:20:45 INFO - jmeter.JMeter: os.version=6.1
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org