You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@trafficserver.apache.org by ezko <er...@harmonicinc.com> on 2018/10/23 08:31:59 UTC

ATS 7.1 graceful shutdown - client connections

Hi,
what is the cleanest way to shutdown traffic server in terms of client
connections.
The use case is rolling updates insides k8s. During update all new traffic
is routed to a new instance , but how do we make sure the terminating
instance has had a change to gracefully complete it's client connections. 
Is using 'traffic_ctl server stop' good enough ?
what about keep alive ? 
Thanks.



--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

Re: ATS 7.1 graceful shutdown - client connections

Posted by ezko <er...@harmonicinc.com>.

Hi Phillip,
we used Jermy's suggestion for draining keep-alives with k8s as follows: 

      lifecycle:
          preStop:
            exec:
              command:
              - /bin/bash
              - -c
              - rm /etc/environment;
                traffic_ctl config set
proxy.config.http.keep_alive_enabled_in 0; sleep 10;
                traffic_ctl server stop; sleep 5;
      readinessProbe:
          exec:
            command:
            - cat
            - /etc/environment

so prior to terminating the POD , k8s will run our preStop hook where we do
the following:
1) remove the file so readiness probe starts to fail , and thus K8s stops
sending us new connections
2) set keep_alive_enabled to 0 , so any current keep alive connections are
drained
3) gracefully stop traffic server

we did some limited scope load testing with Jmeter (our use case is live
HLS) and rolling updates passed without errors. 
But interestingly enough they also passed without steps 2) and 3) above. i'm
not sure why (maybe due to some Jmeter keep-alive configuration)

Additionally we tried using nginx ingress controller. this one had some
errors with Jmeter due to:
https://github.com/kubernetes/ingress-nginx/issues/489





--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

Re: ATS 7.1 graceful shutdown - client connections

Posted by Phillip Moore <pd...@pobox.com>.

We have a tool that we use to take nodes in and out of service. Basically
it either creates or removes a server_enabled file. This causes our L4 load
balancer to remove the ATS node from its pool.  In addition to facilitate
draining in ATS header_rewrite plugin config we have this:

# close keepalive sessions if server is disabled
cond %{SEND_RESPONSE_HDR_HOOK}
cond %{ACCESS:/path/to/server_enabled}    [NOT,OR]
set-header Connection "close"

Caveat is this doesn't work with HTTP/2 I'm told, but we don't do that so
this works nicely for us. I can't take credit for the header_rewrite trick,
I believe it came from Yahoo.

--
Phillip

On Mon, Nov 5, 2018 at 7:25 AM ezko <er...@harmonicinc.com> wrote:

> Thanks Jermy,
> this sounds like a much simpler solution.
>
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
>

Re: ATS 7.1 graceful shutdown - client connections

Posted by ezko <er...@harmonicinc.com>.

Thanks Jermy,
this sounds like a much simpler solution.  





--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

Re: ATS 7.1 graceful shutdown - client connections

Posted by Jeremy Payne <jp...@gmail.com>.

Maybe I'm missing something here..

If the main purpose here is to issue a 'Connection:close'
why not just disable client facing http keep-alive ?

traffic_ctl config set proxy.config.http.keep_alive_enabled_in 0

After the param has been activated, requests on existing connections
will receive a Connection:close.

Of course you still need to take your server out of l4/7 lb rotation
or dns load balancing rotation
so subsequent connections don't end up on the 'draining' server.

On Sun, Oct 28, 2018 at 2:30 AM ezko <er...@harmonicinc.com> wrote:
>
> thanks Masaori Koshiba-2
> we will try it.
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/

Re: ATS 7.1 graceful shutdown - client connections

Posted by ezko <er...@harmonicinc.com>.

thanks Masaori Koshiba-2
we will try it.




--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

Re: ATS 7.1 graceful shutdown - client connections

Posted by Masaori Koshiba <ma...@apache.org>.

> for v8.0.0 documentation for drain states:
> This option modifies the behavior of traffic_ctl server restart ...
> So will drain also work for traffic_ctl server stop ?

Oh, docs looks wrong. According to the source, server stop also has --drain
option.

https://github.com/apache/trafficserver/blob/82bbd37f80dec30315c86bbeb8803d73a487387e/src/traffic_ctl/server.cc#L122

> for older version:
> do you recommend to always inject "Connection: close" , or can we trigger
it
> before shutdown ?

We can trigger it before shutdown. Example of this case is here.

# Close Connections for draining
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/plugins/header_rewrite.en.html#close-connections-for-draining

- Masaori

2018年10月24日(水) 23:09 ezko <er...@harmonicinc.com>:

> Thanks.
> a few follow up questions.
> for v8.0.0 documentation for drain states:
> This option modifies the behavior of traffic_ctl server restart ...
> So will drain also work for traffic_ctl server stop ?
>
> for older version:
> do you recommend to always inject "Connection: close" , or can we trigger
> it
> before shutdown ?
> Thanks.
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
>

Re: ATS 7.1 graceful shutdown - client connections

Posted by ezko <er...@harmonicinc.com>.

Thanks.
a few follow up questions.
for v8.0.0 documentation for drain states:
This option modifies the behavior of traffic_ctl server restart ...
So will drain also work for traffic_ctl server stop ?

for older version:
do you recommend to always inject "Connection: close" , or can we trigger it
before shutdown ?
Thanks.



--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

Re: ATS 7.1 graceful shutdown - client connections

Posted by Masaori Koshiba <ma...@apache.org>.

"Drain" option is available from v8.0.0. You can use it by "traffic_ctl
server stop --drain". Details in below.
This works for both of HTTP/1.1 and HTTP/2.

https://docs.trafficserver.apache.org/en/8.0.x/appendices/command-line/traffic_ctl.en.html#cmdoption-traffic-ctl-server-restart-drain

For older version, you can inject "Connection: close" header by
header_rewrite plugin. But please note that this only works for HTTP/1.1.
HTTP/2 graceful shutdown is supported by v8.0.0.

Thanks,
Masaori

2018年10月23日(火) 17:32 ezko <er...@harmonicinc.com>:

> Hi,
> what is the cleanest way to shutdown traffic server in terms of client
> connections.
> The use case is rolling updates insides k8s. During update all new traffic
> is routed to a new instance , but how do we make sure the terminating
> instance has had a change to gracefully complete it's client connections.
> Is using 'traffic_ctl server stop' good enough ?
> what about keep alive ?
> Thanks.
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
>