You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "nilone (JIRA)" <ji...@apache.org> on 2018/11/20 08:21:00 UTC
[jira] [Updated] (IMPALA-7874) The submitted SQL is stuck and
cannot be quickly entered into the execution state.
[ https://issues.apache.org/jira/browse/IMPALA-7874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nilone updated IMPALA-7874:
---------------------------
Description:
Due to the recent tough problems encountered in impala, we may decide to abandon its use in production, This is a great pity. !!! We have used impala for several years and have tried to solve some problems, but this time we have tried a lot of trials and it has no effect.
The system applies impala to ETL data processing. Currently, the main problem is that the SQL task startup delay submitted by the client is in the CREATED state on the Coordinator 25000 web, not the Running state. These tasks cannot be seen quickly on the CM interface.
.It is possible to wait for a while to run past, or it may stay stuck and die for a long time. The situation may seem to be related to metadata loading, related to the catalogd service. We have more than 300,000 tables and 8 million partitions, and have tried to reduce some useless tables, but in the end did not capture slow queries on the Mysql metabase. we've tried to restart Hive ,Namenode.
Later, we found the problem was similar to that described by IMPALA-5058. We tried to upgrade the version of impala to 2.12.0 using overwritten file mode. The service log became more detailed, but it was still not solved. Tracking the log of catalogd found the following rule. Once "Remaining items in queue: 0, Loads in progress: 1" appears in some worker threads, the problem will appear. We tried to analyze the jstack information of catalogd and found that a large number of threads are in Waiting state.
This is our cluster environment:
CDH 5.13.1 impala-2.10.0 & 2.12.0
300 datanodes, 80 impalad
was:
Due to the recent tough problems encountered in impala, we may decide to abandon its use in production, This is a great pity. !!! We have used impala for several years and have tried to solve some problems, but this time we have tried a lot of trials and it has no effect.
The system applies impala to ETL data processing. Currently, the main problem is that the SQL task startup delay submitted by the client is in the CREATED state on the Coordinator 25000 web, not the Running state. These tasks cannot be quickly queried in the CM. As you can see from the list, it is possible to wait for a while to run past, or it may stay stuck and die for a long time. The situation may seem to be related to metadata loading, related to the catalogd service. We have more than 300,000 tables and 8 million partitions, and have tried to reduce some useless tables, but in the end did not capture slow queries on the Mysql metabase. we've tried to restart Hive ,Namenode.
Later, we found the problem was similar to that described by IMPALA-5058. We tried to upgrade the version of impala to 2.12.0 using overwritten file mode. The service log became more detailed, but it was still not solved. Tracking the log of catalogd found the following rule. Once "Remaining items in queue: 0, Loads in progress: 1" appears in some worker threads, the problem will appear. We tried to analyze the jstack information of catalogd and found that a large number of threads are in Waiting state.
This is our cluster environment:
CDH 5.13.1 impala-2.10.0 & 2.12.0
300 datanodes, 80 impalad
> The submitted SQL is stuck and cannot be quickly entered into the execution state.
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-7874
> URL: https://issues.apache.org/jira/browse/IMPALA-7874
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Catalog
> Affects Versions: Impala 2.10.0, Impala 2.12.0
> Reporter: nilone
> Priority: Major
> Attachments: KB)~_BMDT1FM4{~U]YD7N[I.png, ZK$_]9Q8B)~JNZJXZL1CAUC.png, Z}SUTV0%(O(I3GW$K5$LBF8.png, `[TL{)]P0R0)C]2D~D~4FQQ.png, }`W}LD7]`IG$T4]O~MMREAE.png
>
>
> Due to the recent tough problems encountered in impala, we may decide to abandon its use in production, This is a great pity. !!! We have used impala for several years and have tried to solve some problems, but this time we have tried a lot of trials and it has no effect.
> The system applies impala to ETL data processing. Currently, the main problem is that the SQL task startup delay submitted by the client is in the CREATED state on the Coordinator 25000 web, not the Running state. These tasks cannot be seen quickly on the CM interface.
> .It is possible to wait for a while to run past, or it may stay stuck and die for a long time. The situation may seem to be related to metadata loading, related to the catalogd service. We have more than 300,000 tables and 8 million partitions, and have tried to reduce some useless tables, but in the end did not capture slow queries on the Mysql metabase. we've tried to restart Hive ,Namenode.
> Later, we found the problem was similar to that described by IMPALA-5058. We tried to upgrade the version of impala to 2.12.0 using overwritten file mode. The service log became more detailed, but it was still not solved. Tracking the log of catalogd found the following rule. Once "Remaining items in queue: 0, Loads in progress: 1" appears in some worker threads, the problem will appear. We tried to analyze the jstack information of catalogd and found that a large number of threads are in Waiting state.
> This is our cluster environment:
> CDH 5.13.1 impala-2.10.0 & 2.12.0
> 300 datanodes, 80 impalad
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org