loads file metadata twice
[ https://issues.apache.org/jira/browse/IMPALA-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dimitris Tsirogiannis resolved IMPALA-6424.
-------------------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.12.0
Change-Id: Ie41a734493dcea0e36d6b051966f1d0302907dee
Reviewed-on:
[http://gerrit.cloudera.org:8080/9224]
Reviewed-by: Dimitris Tsirogiannis <
[dtsirogiannis@cloudera.com|mailto:dtsirogiannis@cloudera.com]
>
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
1 file changed, 23 insertions(+), 5 deletions(-)
> REFRESH right after invalidate metadata <table> loads file metadata twice
> -------------------------------------------------------------------------
>
> Key: IMPALA-6424
> URL: https://issues.apache.org/jira/browse/IMPALA-6424
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Juan Yu
> Assignee: Dimitris Tsirogiannis
> Priority: Critical
> Fix For: Impala 2.12.0
>
>
> Compare with normal REFRESH, REFRESH right after Invalidate metadata <table> load file metadata twice and takes 2x time. The second refresh seems redundant.
> I0119 07:46:41.107390 26758 CatalogServiceCatalog.java:1518] Invalidating table metadata: s3.catalog_sales
> I0119 07:46:43.002053 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1166
> I0119 07:46:43.002068 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1166
> I0119 07:46:46.696725 26758 TableLoadingMgr.java:70] Loading metadata for table: s3.catalog_sales
> I0119 07:46:46.696781 26758 TableLoadingMgr.java:72] Remaining items in queue: 0. Loads in progress: 1
> I0119 07:46:46.696857 27023 TableLoader.java:58] Loading metadata for: s3.catalog_sales
> I0119 07:46:46.713222 27023 HdfsTable.java:1206] Fetching partition metadata from the Metastore: s3.catalog_sales
> I0119 07:46:46.905102 27023 HdfsTable.java:1210] Fetched partition metadata from the Metastore: s3.catalog_sales
> *I0119 07:46:46.939254 27023 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20*
> I0119 07:47:00.426975 27023 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
> I0119 07:47:00.427062 27023 TableLoader.java:97] Loaded metadata for: s3.catalog_sales
> I0119 07:47:00.427243 26758 CatalogServiceCatalog.java:1433] Refreshing table metadata: s3.catalog_sales
> I0119 07:47:00.441572 26758 HdfsTable.java:1193] Incrementally loading table metadata for: s3.catalog_sales
> *I0119 07:47:00.456437 26758 HdfsTable.java:834] Loading file and block metadata for 1837 paths for table s3.catalog_sales using a thread pool of size 20*
> I0119 07:47:14.038097 26758 HdfsTable.java:874] Loaded file and block metadata for s3.catalog_sales
> I0119 07:47:14.038132 26758 HdfsTable.java:1203] Incrementally loaded table metadata for: s3.catalog_sales
> I0119 07:47:14.038179 26758 CatalogServiceCatalog.java:1456] Refreshed table metadata: s3.catalog_sales
> I0119 07:47:14.062625 26309 catalog-server.cc:331] Publishing update : TABLE:s3.catalog_sales@1168
> I0119 07:47:14.062645 26309 catalog-server.cc:331] Publishing update : CATALOG:b0f520a5e2ab4056:b7e2e045fa39d625@1168
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)