You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2021/10/04 12:40:20 UTC

[GitHub] [trafficcontrol] smalenfant opened a new issue #6248: Origin Max Connection: Don't change origin max connection for server in ADMIN_DOWN state

smalenfant opened a new issue #6248:
URL: https://github.com/apache/trafficcontrol/issues/6248


   From discussions:
   While doing some testing this morning, I notice that the max origin connection handling can produce some unexpected results. For example, we put a mid cache in ADMIN_DOWN and the max connection goes up after a SYNCDS. The problem is that when you change a server back to REPORTED, those max connection don't get back to normal and requires a SYNCDS to be triggered again.
   
   I wonder if it would be better to keep the max origin connection intact while a mid cache server is brought into ADMIN_DOWN.
   
   This mostly doesn't affect anything. But it keeps the max origin connection flipping up and down in the configuration
   
   The ADMIN_DOWN state is usually short lived. 
   
   Proposal: Create a new t3c-apply parameter to ignore max origin connection instead of changing to current behavior.  
   
   ## This Improvement request (usability, performance, tech debt, etc.) affects these Traffic Control components:
   
   - Traffic Control Cache Config (T3C, formerly ORT)
   
   ## Current behavior:
   When a mid cache is set to ADMIN_DOWN, all the max origin connections are changed across the CDN. That mostly requires a manual queue update and SYNCDS. 
   
   ## New behavior:
   When servers are set to ADMIN_DOWN, do not change the max origin connection calculation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [trafficcontrol] rob05c commented on issue #6248: Origin Max Connection: Don't change origin max connection for server in ADMIN_DOWN state

Posted by GitBox <gi...@apache.org>.
rob05c commented on issue #6248:
URL: https://github.com/apache/trafficcontrol/issues/6248#issuecomment-933544818


   I don't think I agree that it's a bug. The Max Origin Connections on a DS is the total number of connections the CDN should send to the Origin, i.e. it should be what the Origin thinks it can handle.
   
   Respecting down state gives us a more accurate number. It means that when you mark a Mid down, the other Mids will increase their number, so the total CDN number remains what it was set on the DS. That could matter. 
   
   Ideally, CDN operators shouldn't hit the Max under normal operation. Hitting it means giving errors to clients; it exists to protect the Origin. But, if you happen to be close to the limit for some reason, marking Mids down may shift more unique traffic to the individual Mids, so if that number is artificially lower, it could potentially cause it to start getting hit and giving clients errors.
   
   So, as it is, until you Syncds after bringing a mid Reported, the number is artificially low.
   
   The proposal is to always keep it artificially low.
   
   > This mostly doesn't affect anything. But it keeps the max origin connection flipping up and down in the configuration
   
   Is the only gain here consistency in the config? I guess I can see that as a small benefit. But the cost of making the MaxConns worse than they could be seems to outweigh that small benefit, to me. As it is, it's sometimes-wrong (when you don't Queue after Snapping a Mid Reported). But changing it to not consider ADMIN_DOWN would make it always-wrong.
   
   I'd also note, ADMIN_DOWN is _supposed_ to be transient, but on our production CDN, it's pretty common for caches to end up that way for extended periods and not get set OFFLINE, just because we have so many caches and so much data it's impossible to keep it all perfect. When that happens, not considering ADMIN_DOWN would leave the Max Conns permanently higher than they should be.
   
   All that said, I'm not completely opposed, if other people feel strongly, and/or there are benefits I'm not seeing aside from not changing the config as much?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [trafficcontrol] ocket8888 commented on issue #6248: Origin Max Connection: Don't change origin max connection for server in ADMIN_DOWN state

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on issue #6248:
URL: https://github.com/apache/trafficcontrol/issues/6248#issuecomment-933605731


   Oh, I misunderstood.
   
   A cache server that's `ADMIN_DOWN` is not meant to serve any traffic, so it shouldn't prevent a DS's Max Origin Connections from being faithfully realized. But if it's not serving traffic, then does the cache server's own maximum connections need to be reduced? Maybe the other cache servers could still have theirs increased without touching the `ADMIN_DOWN` cache server's connections? I think that would have the potential for windows of operations where the max could be exceeded, but that seems unavoidable in any case when bringing the `ADMIN_DOWN` server back to `REPORTED`/`ONLINE`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [trafficcontrol] rob05c commented on issue #6248: Origin Max Connection: Don't change origin max connection for server in ADMIN_DOWN state

Posted by GitBox <gi...@apache.org>.
rob05c commented on issue #6248:
URL: https://github.com/apache/trafficcontrol/issues/6248#issuecomment-933619958


   > But if it's not serving traffic, then does the cache server's own maximum connections need to be reduced?
   
   It shouldn't matter, the cache won't be serving traffic as soon as the queue/snap propagates.
   
   > Maybe the other cache servers could still have theirs increased without touching the ADMIN_DOWN cache server's connections?
   
   It seems like you're talking about the very short window during config propagation, that the DOWN server is still receiving traffic, but other caches have had their max increased?
   
   I don't think that's the concern with this issue, unless I misunderstand. 
   
   Personally I'm not really concerned about that; unless other people are? It's a pretty short window, and it seems equally likely the cache will drain at the same speed other caches get the higher number. It seems likely it amortizes out to pretty close to the same number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [trafficcontrol] ocket8888 commented on issue #6248: Origin Max Connection: Don't change origin max connection for server in ADMIN_DOWN state

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on issue #6248:
URL: https://github.com/apache/trafficcontrol/issues/6248#issuecomment-933532203


   > Proposal: Create a new t3c-apply parameter to ignore max origin connection instead of changing to current behavior.
   
   This seems like a bug, why add a Parameter instead of just fixing it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [trafficcontrol] ocket8888 commented on issue #6248: Origin Max Connection: Don't change origin max connection for server in ADMIN_DOWN state

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on issue #6248:
URL: https://github.com/apache/trafficcontrol/issues/6248#issuecomment-933659791


   hmm I think I just don't know enough about how this is implemented to offer any valuable suggestions. I'm gonna bow out here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org