Dealing with large files: just throw it away?

[Dale Ghent <da...@elemental.org>]

Hi all,

I'm poking the list here as a relatively novice jmeter user look for some advice, as my google-fu has failed me and I can't seem to find a straight-forward solution to the predicament that I've ran into. I hope I haven't missed something obvious, and kind pointers to the right place are appreciated if I have.

I'm trying to simulate a full application-API conversation in order to load-rate the infrastructure that serves the API. One of the requests in the conversation involves fetching a ca. 1GB file and my only interest is to see how long it takes to download and other such basic connection statistics. I am not interested in its contents (it's just a static binary file), or anything of the sort. As soon as any bytes are received, the relevant counters should be updated and those bytes deallocated or nulled out into the oblivion, never to occupy memory again.

The seasoned among you might already identify the problem I'm running into: jmeter quickly croaks while ingesting this large file with a OOM exception, as the default HTTP sampler appears to be trying its best to stuff the whole thing into memory as its bits come in off the wire. Understandable, given that I suppose it queues up the information for any post-processors to consume, but in this case there aren't any and never will be.

I guess what I'm looking for is essentially a "null" HTTP sampler - where ingested bytes are immediately tossed after they're accounted for. No stuffing into byte arrays or filestreams or anything of the sort. I *hope* this would be a solution to my problem, and I acknowledge that the problem may be more nuanced than it appears to me. Anyone have some good ideas?

TIA,
/dale
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Dealing with large files: just throw it away?

[Dale Ghent <da...@elemental.org>]

> On Feb 6, 2020, at 8:03 PM, Andrew Burton <an...@gmail.com> wrote:
> 
> Hi Dale,
> 
> I believe if you enable "Save response as MD5 hash" it will not store the
> result, which is likely where JMeter runs into issues with large files (see
> the comments for this setting in
> https://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request)

Ah, ok. I have to admit I considered this option but reasoned that, to be able to hash the response, it needs to save it in its entirety either in memory on on disk... which is what I was trying to avoid in the first place. It wasn't entirely clear from the documentation that one would be spared from memory issues. Thanks for pointing that out, it does seem to do the trick.

> Out of curiosity, how often are you going to be dishing up these 1Gb files
> in Prod? You might run into issues simulating this, and instead end up
> saturating the network for your load generators. Returning a 1Gb payload
> for an API request sounds like it could open up some real issues (if I'm
> interpreting your use case correctly).

Fair question. We have a range of network types available in our test environment, including 10, 40, and 100Gb ethernet to the servers. The 1GB files aren't being served as part of the actual API itself, but as a part of a larger scenario that involves many devices querying an API for the availability of any software upgrades and then reacting appropriately to what the API informs them.

/dale


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Dealing with large files: just throw it away?

[Andrew Burton <an...@gmail.com>]

Hi Dale,

I believe if you enable "Save response as MD5 hash" it will not store the
result, which is likely where JMeter runs into issues with large files (see
the comments for this setting in
https://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request)

Also:
https://softwaresaints.co.uk/2016/07/14/performance-testing-applications-with-a-large-response-using-jmeter/


Out of curiosity, how often are you going to be dishing up these 1Gb files
in Prod? You might run into issues simulating this, and instead end up
saturating the network for your load generators. Returning a 1Gb payload
for an API request sounds like it could open up some real issues (if I'm
interpreting your use case correctly).

Andrew


On Fri, Feb 7, 2020 at 7:57 AM Dale Ghent <da...@elemental.org> wrote:

>
> Hi all,
>
> I'm poking the list here as a relatively novice jmeter user look for some
> advice, as my google-fu has failed me and I can't seem to find a
> straight-forward solution to the predicament that I've ran into. I hope I
> haven't missed something obvious, and kind pointers to the right place are
> appreciated if I have.
>
> I'm trying to simulate a full application-API conversation in order to
> load-rate the infrastructure that serves the API. One of the requests in
> the conversation involves fetching a ca. 1GB file and my only interest is
> to see how long it takes to download and other such basic connection
> statistics. I am not interested in its contents (it's just a static binary
> file), or anything of the sort. As soon as any bytes are received, the
> relevant counters should be updated and those bytes deallocated or nulled
> out into the oblivion, never to occupy memory again.
>
> The seasoned among you might already identify the problem I'm running
> into: jmeter quickly croaks while ingesting this large file with a OOM
> exception, as the default HTTP sampler appears to be trying its best to
> stuff the whole thing into memory as its bits come in off the wire.
> Understandable, given that I suppose it queues up the information for any
> post-processors to consume, but in this case there aren't any and never
> will be.
>
> I guess what I'm looking for is essentially a "null" HTTP sampler - where
> ingested bytes are immediately tossed after they're accounted for. No
> stuffing into byte arrays or filestreams or anything of the sort. I *hope*
> this would be a solution to my problem, and I acknowledge that the problem
> may be more nuanced than it appears to me. Anyone have some good ideas?
>
> TIA,
> /dale
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>