A doubt about configuring Apache server and ATS on the same machine

["Kanishka ." <ka...@yahoo-inc.com>]

Hi,

  I've been trying to configure an Apache server and ATS on a host. For us, the latency matters more than RPS/QPS.

  We've observing a huge spike in latency for a small number of requests. Typically, the 99.99%-ile are in tens to hundreds of ms. The 100%-ile is often couple of single digit seconds. Where as the 50%-ile is less than 2 ms and 99.9%-ile is nearly 5ms. Above %-iles are all based on observations/measurements from clients side.

  In general, a unloaded server takes around 2 ms to process a request (e.g. Time spent in the Apache module processing the request).

  In terms of RPS, the best so far has been around 2.2K RPS with 16 clients programs and Apache (2.2) server with max client set to 31.

  We're using persistent connections between the client and the ATS and transient/non-persistent connections  between ATS and Apache.

  Apache processes are configured to served unlimited number of requests. It uses preform MPM. [ET_NET 0] uses up to 80% or more cpu (based on top).

  The 99.9%-ile latencies worse if the number of clients exceed the no of Apache processes.

  This is the first time I'm experimenting with ATS.  Pointers towards keeping the latency in check and maximizing the performance without impacting the latency will be highly appreciated. Currently, the max cup utilization is ~25%.

  Please note, I want to use ATS only as a reverse proxy and I don't intend to use any other functionality (like caching etc).

  Thanks,
Kanishka

Re: A doubt about configuring Apache server and ATS on the same machine

[Bryan Call <bc...@yahoo-inc.com>]

You might want to turn on slow log and see where the requests over 10ms are taking time in the request process.

-Bryan



On Jan 17, 2014, at 3:29 PM, Kanishka . <ka...@yahoo-inc.com> wrote:

> Hi,
> 
>  I've been trying to configure an Apache server and ATS on a host. For us, the latency matters more than RPS/QPS.
> 
>  We've observing a huge spike in latency for a small number of requests. Typically, the 99.99%-ile are in tens to hundreds of ms. The 100%-ile is often couple of single digit seconds. Where as the 50%-ile is less than 2 ms and 99.9%-ile is nearly 5ms. Above %-iles are all based on observations/measurements from clients side.
> 
>  In general, a unloaded server takes around 2 ms to process a request (e.g. Time spent in the Apache module processing the request).
> 
>  In terms of RPS, the best so far has been around 2.2K RPS with 16 clients programs and Apache (2.2) server with max client set to 31.
> 
>  We're using persistent connections between the client and the ATS and transient/non-persistent connections  between ATS and Apache.
> 
>  Apache processes are configured to served unlimited number of requests. It uses preform MPM. [ET_NET 0] uses up to 80% or more cpu (based on top).
> 
>  The 99.9%-ile latencies worse if the number of clients exceed the no of Apache processes.
> 
>  This is the first time I'm experimenting with ATS.  Pointers towards keeping the latency in check and maximizing the performance without impacting the latency will be highly appreciated. Currently, the max cup utilization is ~25%.
> 
>  Please note, I want to use ATS only as a reverse proxy and I don't intend to use any other functionality (like caching etc).
> 
>  Thanks,
> Kanishka

Re: A doubt about configuring Apache server and ATS on the same machine

[James Peach <jp...@apache.org>]

On Jan 17, 2014, at 6:29 AM, Kanishka . <ka...@yahoo-inc.com> wrote:

> Hi,
> 
>  I've been trying to configure an Apache server and ATS on a host. For us, the latency matters more than RPS/QPS.
> 
>  We've observing a huge spike in latency for a small number of requests. Typically, the 99.99%-ile are in tens to hundreds of ms. The 100%-ile is often couple of single digit seconds. Where as the 50%-ile is less than 2 ms and 99.9%-ile is nearly 5ms. Above %-iles are all based on observations/measurements from clients side.
> 
>  In general, a unloaded server takes around 2 ms to process a request (e.g. Time spent in the Apache module processing the request).
> 
>  In terms of RPS, the best so far has been around 2.2K RPS with 16 clients programs and Apache (2.2) server with max client set to 31.
> 
>  We're using persistent connections between the client and the ATS and transient/non-persistent connections  between ATS and Apache.
> 
>  Apache processes are configured to served unlimited number of requests. It uses preform MPM. [ET_NET 0] uses up to 80% or more cpu (based on top).
> 
>  The 99.9%-ile latencies worse if the number of clients exceed the no of Apache processes.
> 
>  This is the first time I'm experimenting with ATS.  Pointers towards keeping the latency in check and maximizing the performance without impacting the latency will be highly appreciated. Currently, the max cup utilization is ~25%.
> 
>  Please note, I want to use ATS only as a reverse proxy and I don't intend to use any other functionality (like caching etc).

I think that the right approach for this problem is to remove ATS from the system and verify that the origin deployment is meeting your requirements. Once you are able to determine that, reintroduce ATS and verify again. At this point you ought to be able to measure ant latency introduced by the proxy.

There are various tunables in ATS that can be used to apply timeouts to requests, forcing them to fail rather than taking forever.

J