[jira] [Created] (IMPALA-11729) Investigate and improve impalad startup time

["Csaba Ringhofer (Jira)" <ji...@apache.org>]

Csaba Ringhofer created IMPALA-11729:
----------------------------------------

             Summary: Investigate and improve impalad startup time
                 Key: IMPALA-11729
                 URL: https://issues.apache.org/jira/browse/IMPALA-11729
             Project: IMPALA
          Issue Type: Improvement
            Reporter: Csaba Ringhofer


impalad startup takes several seconds, even few seconds before trying connecting to statestored. From a  test run (release mode) with a parallel catalogd startup:
{code}
I1113 21:02:17.334743  4363 logging.cc:247] stdout will be logged to this file.
I1113 21:02:18.968991  4363 JniFrontend.java:141] Java Input arguments:
I1113 21:02:19.887519  4363 exec-env.cc:467] Starting statestore subscriber service
{code}
After connecting to statestore coordinators need to wait for the initial catalog update and processing it will take time depending on the number of catalog objects:
{code}
I1113 21:02:19.888423  4363 Frontend.java:1618] Waiting for local catalog to be initialized, attempt: 0
I1113 21:02:21.888621  4363 Frontend.java:1618] Waiting for local catalog to be initialized, attempt: 1
I1113 21:02:23.888849  4363 Frontend.java:1614] Local catalog initialized after: 4000 ms.
I1113 21:02:23.890105  4363 impala-server.cc:3103] Impala has started.
{code}

Meanwhile on catalogd it takes 2 seconds before even trying to connect to HMS:
{code}
I1113 21:02:17.289606  4281 logging.cc:247] stdout will be logged to this file.
I1113 21:02:19.023339  4281 HiveMetaStoreClient.java:720] Trying to connect to metastore with URI (thrift://localhost:9083) in binary transport mode
I1113 21:02:21.671665  5028 catalog-server.cc:400] A catalog update with 1647 entries is assembled. Catalog version: 1649 Last sent catalog version: 0
{code}

While this 6 secs at impalad with ~2 secs waiting for initial catalog update is not very bad, making it quicker would be visible in test run times (custom cluster tests restart the cluster a lot) and in autoscaling scenarios. Finding out what takes the time during startup would be also nice ramp up task.

The startup logic is single threaded - I see the most potential in moving some independent tasks to separate threads. It is also possible that we are doing some completely unnecessary tasks in some scenarios (e..g executor only impalad) or that some tasks could be safely moved to a later point when they are actually needed,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org