[jira] [Resolved] (IMPALA-12350) Daemon fails to initialize large catalog

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang resolved IMPALA-12350.
-------------------------------------
    Resolution: Workaround

Thank [~saulius.vl]! I'll resove this as workaround provided.

We will track the fix of sending a >2GB initial catalog update in IMPALA-13020. Feel free to file new JRIAs for other issues.

BTW, the slower invalidate/refresh operations might be fixed by IMPALA-11501.

> Daemon fails to initialize large catalog
> ----------------------------------------
>
>                 Key: IMPALA-12350
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12350
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Impala 4.2.0
>            Reporter: Saulius Valatka
>            Priority: Major
>
> When the statestored catalog topic is large enough (>2gb) daemons fail to restart and get stuck in a loop:
> {{I0808 13:07:17.702653 3633556 Frontend.java:1618] Waiting for local catalog to be initialized, attempt: 2068}}
>  
> The statestored reports errors as follows:
> {{I0808 13:07:05.587296 2134270 thrift-util.cc:196] TSocket::write_partial() send() <Host: gs1-hdp-data70 Port: 23000>: Broken pipe}}
> {{I0808 13:07:05.587356 2134270 client-cache.h:362] RPC Error: Client for gs1-hdp-data70:23000 hit an unexpected exception: write() send(): Broken pipe, type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala20TUpdateStateResponseE, send: not done}}
> {{I0808 13:07:05.587365 2134270 client-cache.cc:174] Broken Connection, destroy client for gs1-hdp-data70:23000}}
>  
> If this happens we are forced to restart statestore and thus the whole cluster, meaning that we can't tolerate failure from even a single daemon.
> Interestingly the catalog topic increased significantly after upgrading from 3.4.0 to 4.2.0 - from ~800mb to ~3.4gb. Invalidate/refresh operations also became significantly slower (~10ms -> 5s).
> Probably related to thrift_rpc_max_message_size? but I see the maximum value is 2gb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)