You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2021/02/02 03:29:29 UTC
[GitHub] [trafficserver] mtorluemke opened a new issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
mtorluemke opened a new issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471
Feels similar to https://github.com/apache/trafficserver/issues/7163 and https://github.com/apache/trafficserver/issues/7054
Fix for #7054 was #7056.
Fix for #7163 was #7258
Both of these metrics function like a counter, instead of a guage:
proxy.process.http.current_server_connections
proxy.process.http.current_server_transactions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] c-taylor commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
c-taylor commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-791383801
> This affects both:
> proxy.process.http.current_parent_proxy_connections
> proxy.process.http.current_server_connections
>
> Behaviour is constantly additive. Do not see decrements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848092409
We are running with a couple differences on our internal 9.0.x branch from what is on ASF. With 9.1.x we reset against the ASF branch, so that could be we are starting to see a slow leak.
I am working through differences in HttpTunnel. Just brining over our 9.0 version into our 9.1 build seems to have plugged the leak, but also I may not have waited long enough.
One HttpTunnel difference is that our 9.0 included PR #7386 which did not land in ASF.
There was also a slight difference between what we started with in #7237 and what landed in ASF which remained in our build.
Will continue to try different variations and see how the builds work out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] c-taylor commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
c-taylor commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-776690420
I believe this issue also gives a 'false positive' for `proxy.config.http.per_server.connection.max`.
Trying to confirm this now...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] moseleymark commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
moseleymark commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-842715023
This is all web hosting traffic, so it's got pretty random traffic patterns. All of my ATS nodes are doing reverse proxy to various parts of 2 /24s (on the same network). We control all the origin selection (all Lua talking to our own DNS), but the actual content could be anything from tiny files to large media files, could be static or dynamic. There are no parents at all, just each trafficserver box talking to origins.
I've been ramping up traffic to our 8.1.1 nodes but I'm starting to see a crash few every days with 8.1.1 ("Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE`, similar to https://github.com/apache/trafficserver/issues/4921), so I was trying out 9.0.1. The traffic serving part looked great. It's just the connection leak that's an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-791037514
One difference in our environments is that you use parent proxy. Does you proxy.process.http.current_parent_proxy_connections setting look ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848092409
We are running with a couple differences on our internal 9.0.x branch from what is on ASF. With 9.1.x we reset against the ASF branch, so that could be why we are starting to see a slow leak.
I am working through differences in HttpTunnel. Just bringing over our 9.0 version into our 9.1 build seems to have plugged the leak, but also I may not have waited long enough.
One HttpTunnel difference is that our 9.0 included PR #7386 which did not land in ASF.
There was also a slight difference between what we started with in #7237 and what landed in ASF which remained in our build.
Will continue to try different variations and see how the builds work out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-799157123
It doesn't seem a like stat issue. According to memory usage dump (SIGUSR1), `Http1ServerSession` is leaking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-844707663
I reverted #7237 and #7336 then the stats got steady. I think the cause is #7237 but I had to revert #7336 as well because it's an additional fix for #7237 (there's dependency).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-847938362
Interesting. I'll work with #7237.
I just noticed on a machine running 9.1.x a little bit of a server session leak. It had been running for 8 hours. Took it out of rotation, and still the proxy.process.http.current_server_connections is around 1000 even though according to the OS only about 30 connections are open to the origin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-833108526
The clearing of the server_session in HttpSM::kill_this is concerning. As discussed in the "Cleaning up" session of my notes on the H2 to origin PR notes, there are multiple references to the server_session in the tunnel, server_entry, and in the server_session member variable. With the H2 to origin keeping track of all of that was made even worse by the possibility of two origin classes. Any kind of characterization of the leaking transaction scenario would be useful. It is quite likely that we are leaking too, but given our transaction distribution not at such a high rate.
https://docs.google.com/document/d/1sK6AyR-1K795NHsJqpRp4pjZRJ7T5KM78uJlRFMpKiI/edit?usp=sharing
I'll think some more if we can backport a more systematic means of ensuring the server_session has been freed by the time we get to HttpSM::kill_this. Just calling do_io_close against the server_session object in HttpSM::kill_this is unlikely to work since there are many cases where the HttpServerSession object has already been freed (via the tunnel or the server_entry).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ywkaras closed issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
ywkaras closed issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848309373
I removed the changes in from PR #7386 from our 9.0.x build, and now that test build is showing some origin session leaking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] c-taylor commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
c-taylor commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-774189397
This affects both:
proxy.process.http.current_parent_proxy_connections
proxy.process.http.current_server_connections
Behaviour is constantly additive. Do not see decrements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-833105652
Do you notice that file descriptors are not being closed as well as server sessions being leaked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-801490819
> isn't that a configuration bug? Why would you set inactive timeout to zero?
There is a couple of lines set it to 0. And as you can see `cancel_inactivity_timeout()` set it to 0 too.
```
$ git grep "set_inactivity_timeout(0)"
iocore/net/QUICNetVConnection.cc: set_inactivity_timeout(0);
iocore/net/QUICNetVConnection.cc: this->set_inactivity_timeout(0);
iocore/net/QUICNetVConnection.cc: this->set_inactivity_timeout(0);
iocore/net/UnixNetVConnection.cc: set_inactivity_timeout(0);
iocore/net/UnixNetVConnection.cc: set_inactivity_timeout(0);
proxy/PluginVC.cc: set_inactivity_timeout(0);
proxy/http2/Http2Stream.cc: set_inactivity_timeout(0);
```
But I'm not disagreeing that the issue is caused by misconfiguration. There might be something that saves the infinite inactivity timeout case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich removed a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich removed a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848888074
[approve ci autest]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] moseleymark edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
moseleymark edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-840919253
I've been running 9.0.1 for about 24 hours and seeing this as well. The internal connection count seems to just tick up and up, but the connections don't seem to exist. Running traffic_ctl over and over, I do see it go *down* at times but the overall trend is to grow and grow.
15867 is the pid of traffic_server
> lsof -n -p 15867 | wc -l
106540
> traffic_ctl metric get proxy.process.net.connections_currently_open
proxy.process.net.connections_currently_open 108760
> traffic_ctl metric get proxy.process.http.current_server_connections
proxy.process.http.current_server_connections 160043
> traffic_ctl metric get proxy.process.current_server_connections
proxy.process.current_server_connections 159993
We don't use parents so this is normal...
> traffic_ctl metric get proxy.process.http.current_parent_proxy_connections
proxy.process.http.current_parent_proxy_connections 0
> netstat -ant | wc -l
25203
> netstat -antp | grep 15867 | wc -l
8640
So somehow ATS thinks it has 4x the number of connections that the kernel thinks the entire box has and 12x the # of connections that the kernel thinks ATS has.
I'm also seeing a ton more CLOSE_WAITs compared to 8.1.1:
> netstat -ant | grep CLOSE | wc -l
4462
On the other boxes in the same 3-node cluster (servicing the same VIPs and same backend origin sites, so extremely similar workload):
> netstat -ant | grep CLOSE_WAIT | wc -l
39
> netstat -ant | grep CLOSE_WAIT | wc -l
148
All my testing looked good (the throughput is in-line with the other 2 nodes in the cluster; sites load ok; etc), and I wouldn't have even noticed this except that we bumped into the connection limit:
[May 13 10:01:50.185] [ACCEPT 0:8080] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262149, net_connections_throttle=262144
[May 13 10:11:50.188] [ACCEPT 0:8443] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262169, net_connections_throttle=262144
[May 13 10:21:50.197] [ACCEPT 0:8443] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262263, net_connections_throttle=262144
[May 13 10:31:50.198] [ACCEPT 0:8443] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262208, net_connections_throttle=262144
[May 13 10:41:50.199] [ACCEPT 0:8080] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262147, net_connections_throttle=262144
I raised proxy.config.net.connections_throttle but the current count seems to creep up and up. I was scouring the upgrade docs to see if I missed something (which could still be the case) and came upon this ticket, so I figured I'd add some info.
No idea if it's related to this but in 9.0.1, I also see a steady of things like:
20210513.20h26m36s BODY_FACTORY: suppressing 'connect#failed_connect' response for url 'X@'
and
20210513.20h26m41s BODY_FACTORY: suppressing 'connect#failed_connect' response for url ''
My 8.1.1 nodes (same VIP, same workload, same backend origin sites) don't have a single equivalent log entry. They of course have entries for other URLs, but always for valid-ish looking URLs, never '' nor 'X@'. I've yet to find a corresponding log entry for the nginx instance that's fronting ATS. Again, I just mention this in the off chance that it's related to the above connection count issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-791020807
Hmm. Interestingly enough current_server_transactions is broken on my H2 to origin build. Our build based against 9.0.1 is also looking good. Are you actually leaking Http1ServerSession objects?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-801561456
I think https://github.com/apache/trafficserver/commit/fb6ca11c81325dc2fbf50a525cb1be008ccda33d has a bug. Making `keep_alive_no_activity_timeout_out` overridable is fine, but using overridden values after detaching a session from a transaction doesn't make much sense. That behavior means "I want to keep this session in the pool for a shorter/longer time than the default after using it for this transaction".
https://issues.apache.org/jira/browse/TS-863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] moseleymark commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
moseleymark commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-840919253
I've been running 9.0.1 for about 24 hours and seeing this as well. The internal connection count seems to just tick up and up, but the connections don't seem to exist. Running traffic_ctl over and over, I do see it go *down* at times but the overall trend is to grow and grow.
15867 is the pid of traffic_server
# lsof -n -p 15867 | wc -l
106540
# traffic_ctl metric get proxy.process.net.connections_currently_open
proxy.process.net.connections_currently_open 108760
# traffic_ctl metric get proxy.process.http.current_server_connections
proxy.process.http.current_server_connections 160043
# traffic_ctl metric get proxy.process.current_server_connections
proxy.process.current_server_connections 159993
# We don't use parents so this is normal...
# traffic_ctl metric get proxy.process.http.current_parent_proxy_connections
proxy.process.http.current_parent_proxy_connections 0
# netstat -ant | wc -l
25203
# netstat -antp | grep 15867 | wc -l
8640
So somehow ATS thinks it has 4x the number of connections that the kernel thinks the entire box has and 12x the # of connections that the kernel thinks ATS has.
I'm also seeing a ton more CLOSE_WAITs compared to 8.1.1:
# netstat -ant | grep CLOSE | wc -l
4462
On the other boxes in the same 3-node cluster (servicing the same VIPs and same backend origin sites, so extremely similar workload):
# netstat -ant | grep CLOSE_WAIT | wc -l
39
# netstat -ant | grep CLOSE_WAIT | wc -l
148
All my testing looked good (the throughput is in-line with the other 2 nodes in the cluster; sites load ok; etc), and I wouldn't have even noticed this except that we bumped into the connection limit:
[May 13 10:01:50.185] [ACCEPT 0:8080] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262149, net_connections_throttle=262144
[May 13 10:11:50.188] [ACCEPT 0:8443] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262169, net_connections_throttle=262144
[May 13 10:21:50.197] [ACCEPT 0:8443] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262263, net_connections_throttle=262144
[May 13 10:31:50.198] [ACCEPT 0:8443] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262208, net_connections_throttle=262144
[May 13 10:41:50.199] [ACCEPT 0:8080] WARNING: too many connections, throttling. connection_type=ACCEPT, current_connections=262147, net_connections_throttle=262144
I raised proxy.config.net.connections_throttle but the current count seems to creep up and up. I was scouring the upgrade docs to see if I missed something (which could still be the case) and came upon this ticket, so I figured I'd add some info.
No idea if it's related to this but in 9.0.1, I also see a steady of things like:
20210513.20h26m36s BODY_FACTORY: suppressing 'connect#failed_connect' response for url 'X@'
and
20210513.20h26m41s BODY_FACTORY: suppressing 'connect#failed_connect' response for url ''
My 8.1.1 nodes (same VIP, same workload, same backend origin sites) don't have a single equivalent log entry. They of course have entries for other URLs, but always for valid-ish looking URLs, never '' nor 'X@'. I've yet to find a corresponding log entry for the nginx instance that's fronting ATS. Again, I just mention this in the off chance that it's related to the above connection count issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-800886872
This is just a possibility, but I thought it's worth noting.
Seems like removal of sessions from the pool depends on inactivity timeout of sessions (or underlying netvcs). If the timeout didn't happen on a session, the session would stay in the pool forever, unless other controls (e.g. `per_server.connection.max`) kick in.
This line below is suspecious, because it relies on the inactive timeout value currently set. If inactivity timeout is cancelled or set to 0, it does nothing. I think we need to read the original value from the config.
https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/proxy/http/HttpSessionManager.cc#L215
https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/iocore/net/P_UnixNetVConnection.h#L348-L352
https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/iocore/net/UnixNetVConnection.cc#L1354-L1360
https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/iocore/net/P_UnixNetVConnection.h#L362-L368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-794750539
What a mess.
```
$ git grep http_current_server_transactions_stat | grep MENT_DYN
proxy/http/Http1ServerSession.cc: HTTP_DECREMENT_DYN_STAT(http_current_server_transactions_stat);
proxy/http/HttpSM.cc: HTTP_DECREMENT_DYN_STAT(http_current_server_transactions_stat);
proxy/http/HttpSM.cc: HTTP_DECREMENT_DYN_STAT(http_current_server_transactions_stat);
proxy/http/HttpSM.cc: HTTP_INCREMENT_DYN_STAT(http_current_server_transactions_stat);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-799157123
It doesn't seem like a stat issue. According to memory usage dump (SIGUSR1), `Http1ServerSession` is leaking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848226643
I think PR #7386 is the difference. I added that to our 9.1.x build. Without that change, we leaked around 6000 origin sessions over the period of a day. With the change, I ran it for an hour or so, then took it out of rotation. The output of the following commands matched each other nicely as the system drained.
```
sudo lsof -iTCP -n -P | grep MAIN | grep -v LISTEN | grep -v ":443->" | grep -v ":80->" | wc -l
traffic_ctl metric match http.current.server_connections
```
I will reopen PR #7386.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-791011922
Interesting, in our 9.0.x build both look ok. From a server running at low time.
```
proxy.process.http.current_server_transactions 179
proxy.process.http.current_server_connections 526
```
I'll double check to make sure we have everything pushed forward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-820929168
#7618 didn't work for this.
I checked the number of sessions in session pool by using #7627, and the number seemed reasonable. It didn't keep going up at minimum. So the leak probably happens outside of the pool and there should be a case that doesn't release (or close) server sessions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848092409
We are running with a couple differences on our internal 9.0.x branch from what is on ASF. With 9.1.x we reset against the ASF branch, so that could be we are starting to see a slow leak.
I am working through differences in HttpTunnel. Just bringing over our 9.0 version into our 9.1 build seems to have plugged the leak, but also I may not have waited long enough.
One HttpTunnel difference is that our 9.0 included PR #7386 which did not land in ASF.
There was also a slight difference between what we started with in #7237 and what landed in ASF which remained in our build.
Will continue to try different variations and see how the builds work out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-801561456
I think https://github.com/apache/trafficserver/commit/fb6ca11c81325dc2fbf50a525cb1be008ccda33d has a bug. Making `keep_alive_no_activity_timeout_out overridable` is fine, but using overridden values after detaching a session from a transaction doesn't make much sense. That behavior means "I want to keep this session in the pool for a shorter/longer time than the default after using it for this transaction".
https://issues.apache.org/jira/browse/TS-863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848888074
[approve ci autest]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-828783424
Just making a note that 9.1.x is not showing problems with the current_* metrics in our production environment. But we didn't notice problems with 9.0.1 either. We aren't using the parent selection logic through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-842693254
@moseleymark what is your traffic like? Our traffic is a mix of pure proxy and smaller cached objects, and we're not seeing the problem. If it is not parent selection based, then presumably it is due to differences in traffic patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] maskit commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
maskit commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-848410871
I've been running a test build with the change from #7386 for an hour, and I don't see the leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-791020807
Hmm. Interestingly enough current_server_transactions is broken on my H2 to origin build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] zwoop commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
zwoop commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-774189819
proxy.process.http.current_server_connections is a derivative metric too, which derives from
```
{"proxy.process.current_server_connections",
RECD_INT,
{"proxy.process.http.current_server_connections", "proxy.process.http.current_parent_proxy_connections"}}};
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] vmamidi commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
vmamidi commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-801315273
> This is just a possibility, but I thought it's worth noting.
>
> Seems like removal of sessions from the pool depends on inactivity timeout of sessions (or underlying netvcs). If the timeout didn't happen on a session, the session would stay in the pool forever, unless other controls (e.g. `per_server.connection.max`) kick in.
>
> This line below is suspecious, because it relies on the inactive timeout value currently set. If inactivity timeout is cancelled or set to 0, it does nothing. I think we need to read the original value from the config.
isn't that a configuration bug? Why would you set inactive timeout to zero?
> https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/proxy/http/HttpSessionManager.cc#L215
>
> https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/iocore/net/P_UnixNetVConnection.h#L348-L352
>
> https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/iocore/net/UnixNetVConnection.cc#L1354-L1360
>
> https://github.com/apache/trafficserver/blob/d4fc16f64d3104d3682d496e47f2a53511fa77e0/iocore/net/P_UnixNetVConnection.h#L362-L368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7471: ATS 9.0.1: current_server_connections, current_server_transactions metrics broken
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7471:
URL: https://github.com/apache/trafficserver/issues/7471#issuecomment-791011922
Interesting, in our 9.0.x build both look ok. From a server running at low time.
```
proxy.process.http.current_server_transactions 179
proxy.process.http.current_server_connections 526
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org