Latency Between Apache And Tomcat

[Azitabh Ajit <az...@gmail.com>]

We have a setup where we have a Elastic LB that spreads load over two
Apache servers A1 and A2. These apache servers render a few php pages and
primarily redirects the API requests to tomcat appplication servers T1 and
T2 as per the following diagram:

    Incoming Request
        |
        |
        |
        \/
        LB
        /\
       /  \
      /    \
    A1     A2
    |\     /|
    | \   / |
    |  \ /  |
    |  / \  |
    T1      T2

We have recently started to note delays between apache and tomcat. Here is
an example log lines from apache mod_slow log and tomcat access log for the
same request:

    APACHE_MOD_SLOW: VNSdtwoAAJkAACnXb-cAAACJ [06/Feb/2015:16:25:51 +0530]
elapsed: 50.58 cpu: 0.00(usr)/0.00(sys) pid: 10711 ip: 10.0.0.153 host:
www.example.com:443 reqinfo: GET /data/v1/url?url=test-508324 HTTP/1.1

    TOMCAT: [06/Feb/2015:16:26:42 +0530] "GET /data/v1/url?url=test-508324
HTTP/1.1" 200 65 10

Apache says the incoming request came at `06/Feb/2015:16:25:51 +0530` and
it took ``50s`` to process the request. Whereas tomcat says it took only
``10ms`` to process the request whereas it received the request at
``06/Feb/2015:16:26:42 +0530``.

It means it took nearly 50s for apache to connect and send the whole
request to tomcat. Apache is using ``mod_proxy_ajp`` to connect to apache.
  Here is the configuration:

    <Proxy balancer://prod>
       BalancerMember ajp://127.0.0.1:8009 route=jvmRoute-8009
connectiontimeout=1 retry=300
       BalancerMember ajp://10.0.0.153:8009 route=jvmRoute-8009
connectiontimeout=1 retry=300
       ProxySet lbmethod=byrequests
    </Proxy>

Here is the connector configuration from tomcat:

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
maxThreads="4096" minSpareThreads="25" maxSpareThreads="75"/>

As per connectiontimeout value, I am assuming apache shouldn't take more
than 1 sec to establish connection. Since both apache and tomcat are both
on the same machine, there shouldn't be much time lag once connection is
established.

If it helps, we are using ``https`` requests. But that, I don'y think has
anything to do with this. We have done ``ab`` test in order to compare
performance using ``https``, ``http`` and connecting tomcat directly. Here
are the stats:

    ab -n5000 -c5 https://example.com/test/100001


    Requests per second:    13.67 [#/sec] (mean)

    Time per request:       365.851 [ms] (mean)

    Time per request:       73.170 [ms] (mean, across all concurrent
requests)
    Transfer rate:          79.96 [Kbytes/sec] received


    Connection Times (ms)
                  min  mean[+/-sd] median   max

    Connect:      236  267  95.5    247    3401

    Processing:    83   98  58.6     89    1959

    Waiting:       82   96  57.5     87    1959

    Total:        319  365 134.0    338    3571


    Percentage of the requests served within a certain time (ms)

      50%    338
      66%    347
      75%    356
      80%    364
      90%    399
      95%    477
      98%    689
      99%    869
     100%   3571 (longest request)

    ab -n5000 -c5 http://example.com/test/100001


    Time per request:       186.015 [ms] (mean)
    Time per request:       37.203 [ms] (mean, across all concurrent
requests)
    Transfer rate:          155.55 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:       74   79  33.4     76    1278
    Processing:    83  107  82.3     91    3964
    Waiting:       82  105  60.9     89     940
    Total:        157  186  90.1    168    4042

    Percentage of the requests served within a certain time (ms)
      50%    168
      66%    174
      75%    180
      80%    184
      90%    211
      95%    259
      98%    379
      99%    507
     100%   4042 (longest request)

    ab -n5000 -c5 http://IP:8080/test/100001


    Requests per second:    31.32 [#/sec] (mean)
    Time per request:       159.624 [ms] (mean)
    Time per request:       31.925 [ms] (mean, across all concurrent
requests)
    Transfer rate:          181.30 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:       71   76  68.4     73    3079
    Processing:    78   84  13.1     81     594
    Waiting:       77   83   6.5     81     185
    Total:        149  159  71.2    154    3313

    Percentage of the requests served within a certain time (ms)
      50%    154
      66%    157
      75%    160
      80%    161
      90%    166
      95%    171
      98%    177
      99%    189
     100%   3313 (longest request)

Following observation leads me to believe this that bad performance is
depending on sequence of events because none of the requests in the test
performed that badly.

Versions:
    Apache: 2.2.4
    Tomcat: 8.0.16


Any ideas where the lag is coming from and how to reduce the  same?