You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/11/15 01:17:00 UTC
[jira] [Commented] (IMPALA-11721) Impala query keep being retried over frequently updated iceberg table
[ https://issues.apache.org/jira/browse/IMPALA-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634100#comment-17634100 ]
ASF subversion and git services commented on IMPALA-11721:
----------------------------------------------------------
Commit af46585801387a41153d4a957336d6f0101d09f7 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=af4658580 ]
IMPALA-11721: Impala query keep being retried over frequently updated iceberg table
Iceberg table loading can fail in local catalog mode if the table gets
updated frequently. This is what happens during table loading in local
catalog mode: every query starts with it's own empty local catalog.
Table metadata is fetched in multiple requests via a MetaProvider which
is always a CatalogdMetaProvider. CatalogdMetaProvider caches requests
and the cache key also includes the table's catalog version.
The Iceberg table is loaded by the following requests:
1 CatalogdMetaProvider.loadTable()
2 CatalogdMetaProvider.loadIcebergTable()
3 CatalogdMetaProvider.loadIcebergApiTable() # This actually directly
# loads the Iceberg table
# via Iceberg API
# (no CatalogD involved)
4 CatalogdMetaProvider.loadTableColumnStatistics()
5 CatalogdMetaProvider.loadPartitionList()
6 CatalogdMetaProvider.loadPartitionsByRefs()
Steps 1-4 happens during table loading, steps 5-6 happens during
planning. We cannot really reorder these invocations, but since
CatalogdMetaProvider caches these, only the very first invocations need
to reach out to CatalogD and check the table's catalog version.
Subsequent invocations, i.e. subsequent queries that use the Iceberg
table can use the cached metadata, and no need to check the catalog
version of the cached metadata since the cache key also includes
the catalog version, hence we have corresponding metadata in the cache.
This patch resolves the issue by pre-warming the metaprovider's cache
before issuing loadIcebergApiTable() so the CatalogdMetaProvider.load*()
operations can be served from cache.
So what happens when the metaprovider's cache gets invalidated due to
concurrent updates to the table and we are still processing the query?
No problem, only the top-level TableCacheKey gets invalidated. The
cache will still be able to answer the fine-grained load requests that
are keyed by the now outdated catalog version. E.g. ColStatsCacheKey
hashes db name, table name, catalog version, and column name as a key
in the cache. Therefore the current query processing can be finished
using a consistent state of the metadata. Subsequent queries will use
a newer version of the table.
Testing:
* modified test_insert_stress.py so it won't tolerate inconsistent
metadata fetch exceptions (Frontend already tolerates them
to some degree)
Change-Id: Iac28224b2b6d67725eeb17f3e9d813ba622edb43
Reviewed-on: http://gerrit.cloudera.org:8080/19234
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Impala query keep being retried over frequently updated iceberg table
> ---------------------------------------------------------------------
>
> Key: IMPALA-11721
> URL: https://issues.apache.org/jira/browse/IMPALA-11721
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Iceberg table loading can fail in local catalog mode if the table gets updated frequently.
> This is what happens during table loading in local catalog mode:
> Every query starts with it's own empty local catalog. Table metadata is fetched in multiple requests via a MetaProvider which is always a CatalogdMetaprovider. CatalogdMetaprovider caches requests and the cache key also includes the table's catalog version.
> The Iceberg table is loaded by the following requests:
> # CatalogdMetaProvider.loadTable()
> # CatalogdMetaProvider.loadIcebergTable()
> # CatalogdMetaProvider.loadIcebergApiTable() # This actually directly loads the Iceberg table via Iceberg API (no CatalogD involved)
> # CatalogdMetaProvider.loadTableColumnStatistics()
> # CatalogdMetaProvider.loadPartitionList()
> # CatalogdMetaProvider.loadPartitionsByRefs()
> Steps 1-4 happens during table loading, steps 5-6 happens during planning. We cannot really reorder these invocations, but since CatalogdMetaprovider caches these, only the very first invocations need to reach out to CatalogD and check the table's catalot version. Subsequent invocations, i.e. subsequent queries that use the Iceberg table can use the cached metadata, and no need to check the catalog version of the cached metadata since the cache key also includes the catalog version.
> I see two things that could resolve the issue:
> # speedup loadIcebergApiTable()
> ** either by speeding up Iceberg table loading itself
> ** or make the Iceberg API table serializable, so we can fetch it from CatalogD
> # Pre-warm the cache before issuing loadIcebergApiTable()
> ** so the CatalogdMetaProvider.load*() operations can be served from cache
> 1 needs contributions to the Iceberg library
> 2 can be done relatively easily. We just need to pre-invoke loadTableColumnStatistics() and
> FeCatalogUtils.loadAllPartitions() (which invokes loadPartitionList() and loadPartitionsByRefs()) before loadIcebergApiTable(). So when they are needed later they can be served from cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org