JMeter metrics lost in InfluxDB

[Tong Sun <su...@gmail.com>]

Hi,

I'm sending my JMeter test results to InfluxDB, which is in a container in
an Azure VM (in the cloud).

During the test I observed only 5 connection timeouts in the JMeter log to
InfluxDB.
However, from the Grafana dashboard, there are lots and lots of metrics
losts.

Has anyone noticed the same problem before?

- It is an extreme light load I'd say, only 10 concurrent users, doing
things at a slow pace.
- I checked the Azure VM. It has extreme light load during my test as well.

So to me, so many metrics losts is kind of unacceptable.
Any fix please?


[image: image.png]

[image: image.png]

Re: JMeter metrics lost in InfluxDB

[Felix Schumacher <fe...@internetallee.de>]

Apart from the general notes I wrote below, assuming you are using the 
influx plugin you mentioned in the other posts. Have you read 
https://github.com/mderevyankoaqa/jmeter-influxdb2-listener-plugin#important-notes 
? There is mentioned, that after five errors the import will be stopped.

Felix

Am 16.07.22 um 13:20 schrieb Felix Schumacher:
>
> Am 16.07.22 um 00:43 schrieb Tong Sun:
>
>> Hi,
>>
>> I'm sending my JMeter test results to InfluxDB, which is in a 
>> container in an Azure VM (in the cloud).
>>
>> During the test I observed only 5 connection timeouts in the JMeter 
>> log to InfluxDB.
>> However, from the Grafana dashboard, there are lots and lots 
>> of metrics losts.
>>
>> Has anyone noticed the same problem before?
>>
>> - It is an extreme light load I'd say, only 10 concurrent users, 
>> doing things at a slow pace.
>> - I checked the Azure VM. It has extreme light load during my test as 
>> well.
>>
>> So to me, so many metrics losts is kind of unacceptable.
>> Any fix please?
>
> Hi,
>
> I am a bit lost, where you are seeing the missing metrics. Is it in 
> the first picture in throughput and response times? What do you expect 
> to see there?
>
> The pattern in throughput and response time looks like either the 
> receiving side misses to store values. (The load pattern in response 
> time looks to me, as it could be connected by straight lines that 
> match the surrounding lines)
>
> If that is the error you are looking for. Have you looked at the GC 
> values of the receiving server? Are there any longer gaps? The gaps 
> seem to long for GC, though. Where they observable on your local setup?
>
> Have you tried to do a simpler load test (no service involved, local 
> samplers like JSR223 or test sampler, only).
>
> Are your JVMs setup correctly? (You might want to look at the Java 
> Flight Recorder and Mission Control for insights into them)
>
> Felix
>
>>
>>
>> image.png
>>
>> image.png

Re: JMeter metrics lost in InfluxDB

[Tong Sun <su...@gmail.com>]

On Sat, Jul 16, 2022 at 7:20 AM Felix Schumacher wrote:

> Am 16.07.22 um 00:43 schrieb Tong Sun:
>
> Hi,
>
> I'm sending my JMeter test results to InfluxDB, which is in a container in
> an Azure VM (in the cloud).
>
> During the test I observed only 5 connection timeouts in the JMeter log to
> InfluxDB.
> However, from the Grafana dashboard, there are lots and lots of metrics
> losts.
>
> Has anyone noticed the same problem before?
>
> - It is an extreme light load I'd say, only 10 concurrent users, doing
> things at a slow pace.
> - I checked the Azure VM. It has extreme light load during my test as well.
>
> So to me, so many metrics losts is kind of unacceptable.
> Any fix please?
>
> Hi,
>
> I am a bit lost, where you are seeing the missing metrics. Is it in the
> first picture in throughput and response times? What do you expect to see
> there?
>
> The pattern in throughput and response time looks like either the
> receiving side misses to store values. (The load pattern in response time
> looks to me, as it could be connected by straight lines that match the
> surrounding lines)
>

Yep, that's exactly what I meant, the lines should be connected by straight
lines, the breaking-up parts mean missing store values.

If that is the error you are looking for. Have you looked at the GC values
> of the receiving server? Are there any longer gaps? The gaps seem to long
> for GC, though. Where they observable on your local setup?
>
> Have you tried to do a simpler load test (no service involved, local
> samplers like JSR223 or test sampler, only).
>
> Are your JVMs setup correctly? (You might want to look at the Java Flight
> Recorder and Mission Control for insights into them)
>

OK. I do see other of my simple tests having continuous connected straight
lines (which should rule out JVMs setup problem I suppose).
And I'll look into the other methods you mentioned, like looking at the GC
values on the receiving server. TBH, these are all new to me,
if there are any reading materials / links that I can refer to, that'll be
appreciated.


BTW,

I'm not using
https://github.com/mderevyankoaqa/jmeter-influxdb2-listener-plugin#compatibility
which supports InfluxDB v2.x only (mine is 1.8, and it is not supported)

The one that I'm using is
org.apache.jmeter.visualizers.backend.influxdb.HttpMetricsSender,

Maybe it too, after five errors the import will be stopped.
If it does, there should be a log somewhere, I'll check carefully for it
next.

Felix
>
Thanks

Re: JMeter metrics lost in InfluxDB

[Felix Schumacher <fe...@internetallee.de>]

Am 16.07.22 um 00:43 schrieb Tong Sun:

> Hi,
>
> I'm sending my JMeter test results to InfluxDB, which is in a 
> container in an Azure VM (in the cloud).
>
> During the test I observed only 5 connection timeouts in the JMeter 
> log to InfluxDB.
> However, from the Grafana dashboard, there are lots and lots 
> of metrics losts.
>
> Has anyone noticed the same problem before?
>
> - It is an extreme light load I'd say, only 10 concurrent users, doing 
> things at a slow pace.
> - I checked the Azure VM. It has extreme light load during my test as 
> well.
>
> So to me, so many metrics losts is kind of unacceptable.
> Any fix please?

Hi,

I am a bit lost, where you are seeing the missing metrics. Is it in the 
first picture in throughput and response times? What do you expect to 
see there?

The pattern in throughput and response time looks like either the 
receiving side misses to store values. (The load pattern in response 
time looks to me, as it could be connected by straight lines that match 
the surrounding lines)

If that is the error you are looking for. Have you looked at the GC 
values of the receiving server? Are there any longer gaps? The gaps seem 
to long for GC, though. Where they observable on your local setup?

Have you tried to do a simpler load test (no service involved, local 
samplers like JSR223 or test sampler, only).

Are your JVMs setup correctly? (You might want to look at the Java 
Flight Recorder and Mission Control for insights into them)

Felix

>
>
> image.png
>
> image.png