You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jmeter.apache.org by bu...@apache.org on 2018/04/03 14:02:14 UTC

[Bug 62251] New: TextGraphiteMetricsSender.writeAndSendMetrics will not invalidate lost connections

https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

            Bug ID: 62251
           Summary: TextGraphiteMetricsSender.writeAndSendMetrics will not
                    invalidate lost connections
           Product: JMeter
           Version: 3.3
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Main
          Assignee: issues@jmeter.apache.org
          Reporter: kecskemetizsolt@gmail.com
  Target Milestone: ---

We are running load tests continuously from multiple instances against multiple
services and the results are sent to hostedgraphite.com using Backend Listener
component with TextGraphiteMetricsSender.

From time to time we experience an issue where metrics stop being sent to
hostedgraphite.com. Sometimes it is just a few instances but occasionally all
of them appear to lose connection to HG.

With enabled debug logging we still see 'Wrote x metrics' in the logs from
TextGraphiteMetricsSender.writeAndSendMetrics() and we don't see any errors or
warns that would be logged in case of exceptions.

We have other applications sending data to HG, and those did not experience a
similar outage at the same time, so we are assuming the issue is with JMeter.

After experimenting with PickleGraphiteMetricsSender we think we found the
reason for this issue.

For the test using PickleGraphiteMetricsSender we see the following logs when
such a loss of connection occurs:

2018-04-03 11:36:17,880 ERROR o.a.j.v.b.g.PickleGraphiteMetricsSender: Error
writing to Graphite: Broken pipe (Write failed)
2018-04-03 11:36:17,880 DEBUG o.a.j.v.b.g.PickleGraphiteMetricsSender: Wrote
444 metrics

The connection is then reestablished and data appears correctly in HG.

In the same JMeter process we have another BackendListener that is using
TextGraphiteMetricsSender that does not log any errors:

2018-04-03 11:36:17,884 DEBUG o.a.j.v.b.g.TextGraphiteMetricsSender: Wrote 66
metrics

This seems to come down to the use of PrintWriter in TextGraphiteMetricsSender.
PrintWriter does not throw exceptions:
https://docs.oracle.com/javase/8/docs/api/java/io/PrintWriter.html.

So when the connection fails in TextGraphiteMetricsSender it is never
invalidated.

PickleGraphiteMetricsSender does not use PrintWriter, it uses
OutputStreamWriter directly.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

--- Comment #4 from Zsolt Kecskemeti <ke...@gmail.com> ---
Not yet, sorry, hopefully I will get a chance to test it tomorrow.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

Philippe Mouawad <p....@ubik-ingenierie.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |FixedInTrunk
   Target Milestone|---                         |JMETER_5.0

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

--- Comment #5 from Zsolt Kecskemeti <ke...@gmail.com> ---
Sorry for the delay.

I had to setup Graphite in docker and break connection using iptables, but just
didn't have the time to do it properly for ages.  

This is fixed now, the bad connection is invalidated correctly in
TextGraphiteMetricsSender as well.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

Philippe Mouawad <p....@ubik-ingenierie.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|3.3                         |2.13
            Summary|TextGraphiteMetricsSender.w |TextGraphiteMetricsSender
                   |riteAndSendMetrics will not |does not invalidate lost
                   |invalidate lost connections |connections in case of
                   |                            |network errors
                 CC|                            |p.mouawad@ubik-ingenierie.c
                   |                            |om

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

Philippe Mouawad <p....@ubik-ingenierie.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #1 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Author: pmouawad
Date: Thu Apr  5 11:03:13 2018
New Revision: 1828416

URL: http://svn.apache.org/viewvc?rev=1828416&view=rev
Log:
Bug 62251 - TextGraphiteMetricsSender does not invalidate lost connections in
case of network errors
Bugzilla Id: 62251

Modified:
   
jmeter/trunk/src/components/org/apache/jmeter/visualizers/backend/graphite/TextGraphiteMetricsSender.java
    jmeter/trunk/xdocs/changes.xml

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

--- Comment #3 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Did you have any opportunity to test nightly build ?

Thanks for your feedback

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 62251] TextGraphiteMetricsSender does not invalidate lost connections in case of network errors

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62251

--- Comment #2 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Thanks for detailed report, can you test next build which will be available
within 30 minutes from now and confirm wether issue is fixed.

Thanks

-- 
You are receiving this mail because:
You are the assignee for the bug.