[jira] [Commented] (IMPALA-14801) Catalog topic update creation can't skip Iceberg tables

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-14801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080046#comment-18080046 ] 

ASF subversion and git services commented on IMPALA-14801:
----------------------------------------------------------

Commit fc78ecb81d496581380d10ae749909d53c073dca in impala's branch refs/heads/master from Mihaly Szjatinya
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fc78ecb81 ]

IMPALA-14801: Catalog topic update creation can't skip Iceberg tables

This patch extends the skipping mechanism from IMPALA-6671 for Iceberg
tables. It uses the Iceberg table's own tableLock, and the
lastVersionSeenByTopicUpdate mechanism of the underlying HDFS table.

Testing:
Added TestTopicUpdateFrequency::test_topic_updates_*_iceberg in 4
variants

Fixed the original HDFS tests:
1. test_topic_updates_unblock
  - By default it was running in Local Catalog mode, which has no effect
    for fast DML/DQL queries. Added variants for both Legacy and Local
    Catalog mode to demonstrate the case.
  - In the blocking scenario there was a missing assert (!), making the
    test always pass.
  - Fast queries are much faster than stated, which doesn't seem to
    matter however for the nature of the test.
  - Reduced query delay times

2. test_topic_updates_advance
  - The test claimed to test catalog_max_lock_skipped_topic_updates but
    experimentally I could see no counter blockings triggered at all
    under any configuration.
  - Added more parallel threads and reduced
    catalog_max_lock_skipped_topic_updates to 2 to reliably trigger the
    blockings. Ran 20 times locally to verify.
  - Query execution time expectations largely incorrect. Perhaps for
    some reason it changed over time.
  - Assert expects the max query time to be no more than a predictable
    value. Which conceptually makes sense for SYNC_DDL, but I wasn't
    able to reliably reproduce the case yet.
  - Hence for now at least checking the blockings occurring in catalog
    logs.
  - Reduced query delay times

3. Removed test_topic_lock_timeout_disabled, it is now covered by one of
   the test_topic_updates_unblock variants

Change-Id: I51e46820aaa096f3eb69f4dcf580e49a69d6603d
Reviewed-on: http://gerrit.cloudera.org:8080/24243
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Csaba Ringhofer <cs...@cloudera.com>


> Catalog topic update creation can't skip Iceberg tables
> -------------------------------------------------------
>
>                 Key: IMPALA-14801
>                 URL: https://issues.apache.org/jira/browse/IMPALA-14801
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>            Reporter: Csaba Ringhofer
>            Assignee: Mihaly Szjatinya
>            Priority: Critical
>
> https://github.com/apache/impala/blob/31769a7fb50ae1d6b6d69d366a776df441e00e3a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L1819
> lockTableAndAddToCatalogDelta() calls lockHdfsTblWithTimeout() only for HdfsTable, which means that for IcerbergTable  tbl.takeReadLock() lock is called, which block the catalog topic update collection till it can take the table lock. This can be a serious issue, as loading Iceberg tables can take a significant amount of time.
> The skipping logic was implemented in IMPALA-6671 for Hive tables: https://github.com/apache/impala/commit/2fccd82590d747d834b8be6f3b05bb446d9bac12
> Test for existing skipping logic: https://github.com/apache/impala/blob/31769a7fb50ae1d6b6d69d366a776df441e00e3a/tests/custom_cluster/test_topic_update_frequency.py#L30
> It uses several debug actions to inject delays to reproduce the blocking issue. Testing Iceberg tables should be possible in a similar way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org