[jira] [Created] (IMPALA-5557) Increase and stagger rpc_default_keepalive_time_ms

["Mostafa Mokhtar (JIRA)" <ji...@apache.org>]

Mostafa Mokhtar created IMPALA-5557:
---------------------------------------

             Summary: Increase and stagger rpc_default_keepalive_time_ms
                 Key: IMPALA-5557
                 URL: https://issues.apache.org/jira/browse/IMPALA-5557
             Project: IMPALA
          Issue Type: Sub-task
          Components: Distributed Exec
            Reporter: Mostafa Mokhtar


Queries were still hitting the KDC hard and eventually failed, this is the sequence of events:
# Start running a query
# Impala backends send thousands of TGS_REQ requests to the KDC
# Query would appear to make some progress 
# Some connections succeed others fail with Timeout exceeded waiting to connect
# Then I noticed in the logs that idle connections get kill after 65 seconds
reactor.cc:281] Timing out connection server connection from 10.17.229.14:41804 - it has been idle for 65.0002s
# Query takes about 2 minutes and fail
# By then most connections are released since they were idle for > 65 seconds
# New queries go through the same process again and eventually fail 

In order to reliably run on a large cluster I believe we need to:
# Change the lifetime of idle connections, possibly extend it to ticket lifetime?
# Create new connections before existing ones expire in a staggered fashion to avoid KDC related failures?

This is the flag rpc_default_keepalive_time_ms



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)