You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/03/27 21:57:00 UTC
[jira] [Commented] (IMPALA-11509) Dropping files of Iceberg during table loading may cause Impalad to stuck in infinite loop
[ https://issues.apache.org/jira/browse/IMPALA-11509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705690#comment-17705690 ]
ASF subversion and git services commented on IMPALA-11509:
----------------------------------------------------------
Commit 2c779939dc302be9ee5dd97ddf374bb043040891 in impala's branch refs/heads/master from Andrew Sherman
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2c779939d ]
IMPALA-11509: Prevent queries hanging when Iceberg metadata is missing.
Traditionally table metadata is loaded by the catalog and sent as thrift
to the Impala daemons. With Iceberg tables, some metadata, for example
the org.apache.iceberg.Table, is loaded in the Coordinator at the same
time as the thrift description is being deserialized. If the loading of
the org.apache.iceberg.Table fails, perhaps because of missing Iceberg
metadata, then the loading of the table fails. This can cause an
infinite loop as StmtMetadataLoader.loadTables() waits hopefully for
the catalog to send a new version of the table.
Change some Iceberg table loading methods to throw
IcebergTableLoadingException when a failure occurs. Prevent the hang by
substituting in an IncompleteTable if an IcebergTableLoadingException
occurs.
The test test_drop_incomplete_table had previously been disabled because
of IMPALA-11509. To re-enable this required a second change. The way
that DROP TABLE is executed on an iceberg table depends on which
Iceberg catalog is being used. If this Iceberg catalog is not a Hive
catalog then the execution happens in two parts, first the Iceberg
table is dropped, then the table is dropped in HMS. If this case, if
the drop fails in Iceberg, we should still continue on to perform the
drop in HMS.
TESTING
- Add a new test, originally developed for IMPALA-11330, which tests
failures after deleting Iceberg metadata.
- Re-enable test_drop_incomplete_table().
Change-Id: I695559e21c510615918a51a4b5057bc616ee5421
Reviewed-on: http://gerrit.cloudera.org:8080/19509
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Dropping files of Iceberg during table loading may cause Impalad to stuck in infinite loop
> ------------------------------------------------------------------------------------------
>
> Key: IMPALA-11509
> URL: https://issues.apache.org/jira/browse/IMPALA-11509
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.1.0
> Reporter: Gabor Kaszab
> Assignee: Andrew Sherman
> Priority: Critical
> Labels: iceberg, impala-iceberg
>
> This issues is very similar to https://issues.apache.org/jira/browse/IMPALA-11502. The repro steps are also almost identical, however in this case the folder of the table should be dropped right when the INSERT into starts.
> Repro steps:
> 1) Create the Iceberg table:
> {code:java}
> DROP DATABASE IF EXISTS `drop_incomplete_table` CASCADE;
> CREATE DATABASE `drop_incomplete_table`;
> CREATE TABLE drop_incomplete_table.iceberg_tbl (i int) stored as iceberg
> tblproperties('iceberg.catalog'='hadoop.catalog',
> 'iceberg.catalog_location'='/test-warehouse/drop_incomplete_table');
> {code}
> 2) For this step timing is essential and might require a few try to hit the issue. Try to run INSERT INTO and dropping the HDFS folder at the same time. Manually executing them is fine, this doesn't require scripting.
> {code:java}
> INSERT INTO drop_incomplete_table.iceberg_tbl VALUES (1), (2), (3);
> hdfs dfs -rm -r hdfs://localhost:20500/test-warehouse/drop_incomplete_table
> {code}
> You will notice you hit the issue when Impala shell start to hang. The jstack of the hanging impalad (not the catalogd) will contain this for one of the threads:
> {code:java}
> "Thread-15" #30 prio=5 os_prio=0 tid=0x000000000db2a000 nid=0x56f4 in Object.wait() [0x00007f0e7b59a000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.apache.impala.catalog.ImpaladCatalog.waitForCatalogUpdate(ImpaladCatalog.java:290)
> - locked <0x0000000724f7cdc0> (a java.lang.Object)
> at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:229)
> at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:141)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2001)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1913)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1737)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
> {code}
> Initially, Iceberg tables are created as IncompleteTables and when there is a query on the table, they will be loaded as IcebergTable. For me it seems, that when we run the first query after creating the table, with some timing of dropping the files we can get into a state where the table appears as a "missingTable" in StmtMetadataLoader.loadTable(), however, when a prioritized table load is requested, the Catalog says that the table is already loaded.
> This results the table always appearing as "missingTable" and we never get out of the [while loop|https://github.com/apache/impala/blob/62e20d1ba842a3f27395251c57dea9850f462fc9/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java#L196] in loadTables().
> I managed to repro this using HiveCatalog, but I didn't have luck to repro with non-Iceberg, traditional Hive tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org