You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/08/29 12:45:00 UTC
[jira] [Commented] (IMPALA-8877) CatalogException during stress
test: Table modified while operation was in progress
[ https://issues.apache.org/jira/browse/IMPALA-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918569#comment-16918569 ]
ASF subversion and git services commented on IMPALA-8877:
---------------------------------------------------------
Commit 08ecb6d28cd7794a654f66fe41bb8ae727bc1a3e in impala's branch refs/heads/master from Anurag Mantripragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=08ecb6d ]
IMPALA-8579: Ignore trivial alter table/partition events.
Hive generates certain trivial alter events for eg: change only
"transient_lastDdlTime". This is seen in the alter events
accompanying an INSERT event. MetastoreEventProcessor should ignore
such events as these trivial properties are not used by Impala
and they cause unnecessary invalidates/refreshes. This change will
also potentially reduce the likelihood of IMPALA-8877 as back to back
invalidates after INSERT operations are avoided.
Testing:
Added tests respectively to testPartitionEvents() and
testAlterTableEvent() in MetastoreEventProcessorTest class.
Change-Id: I01a59d5170accc014f76f14eb526d96ddcf61f76
Reviewed-on: http://gerrit.cloudera.org:8080/14145
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> CatalogException during stress test: Table <foo> modified while operation was in progress
> -----------------------------------------------------------------------------------------
>
> Key: IMPALA-8877
> URL: https://issues.apache.org/jira/browse/IMPALA-8877
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.3.0
> Reporter: David Knupp
> Assignee: Vihang Karajgaonkar
> Priority: Critical
> Attachments: catalogd.INFO.tar.gz, impalad.INFO.tar.gz
>
>
> This was hit while running the stress tests to get a baseline on a deployed cluster.
> /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> {noformat}
> Query (id=924a50178a5a6146:29d58a7300000000)
> Summary
> Session ID: 5543fb9029e2b71f:f446381b1f59ed81
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2019-08-19 01:26:07.292866000
> End Time: 2019-08-19 01:26:27.248053000
> Query Type: DDL
> Query State: EXCEPTION
> Query Status: CatalogException: Table 'tpcds_300_decimal_parquet.catalog_sales' was modified while operation was in progress, aborting execution.
> Impala Version: impalad version 3.3.0-SNAPSHOT RELEASE (build df3e7c051e2641524fc53a0cd07c2a14decd55f7)
> User: systest@VPC.CLOUDERA.COM
> Connected User: systest@VPC.CLOUDERA.COM
> Delegated User:
> Network Address: ::ffff:10.65.6.19:39174
> Default Db: tpcds_300_decimal_parquet
> Sql Statement: /* Mem: 12850 MB. Coordinator: quasar-mzmnbe-6.vpc.cloudera.com. */
> COMPUTE STATS catalog_sales
> Coordinator: quasar-mzmnbe-6.vpc.cloudera.com:22000
> Query Options (set by configuration): ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> Query Options (set by configuration and planner): ABORT_ON_ERROR=1,MEM_LIMIT=13474201600,MT_DOP=4,EXEC_TIME_LIMIT_S=2147483647,TIMEZONE=America/Los_Angeles,DEFAULT_FILE_FORMAT=4,DEFAULT_TRANSACTIONAL_TYPE=1
> DDL Type: COMPUTE_STATS
> Query Compilation
> Metadata of all 1 tables cached: 5.62s (5622372318)
> Analysis finished: 5.62s (5622560027)
> Authorization finished (noop): 5.62s (5622568284)
> Retried query planning due to inconsistent metadata 7 of 40 times: Catalog object TCatalogObject(type:TABLE, catalog_version:94204, table:TTable(db_name:tpcds_300_decimal_parquet, tbl_name:catalog_sales)) changed version between accesses.: 5.95s (5949859598)
> Planning finished: 5.95s (5949861145)
> Query Timeline
> Query submitted: 0ns (0)
> Planning finished: 5.95s (5950024020)
> Child queries finished: 17.85s (17849072057)
> Rows available: 19.82s (19825080035)
> Unregister query: 19.95s (19955080560)
> Frontend
> - CatalogFetch.ColumnStats.Misses: 34 (34)
> - CatalogFetch.ColumnStats.Requests: 34 (34)
> - CatalogFetch.ColumnStats.Time: 0 (0)
> - CatalogFetch.Config.Hits: 1 (1)
> - CatalogFetch.Config.Requests: 1 (1)
> - CatalogFetch.Config.Time: 0 (0)
> - CatalogFetch.DatabaseList.Hits: 8 (8)
> - CatalogFetch.DatabaseList.Requests: 8 (8)
> - CatalogFetch.DatabaseList.Time: 0 (0)
> - CatalogFetch.PartitionLists.Misses: 1 (1)
> - CatalogFetch.PartitionLists.Requests: 1 (1)
> - CatalogFetch.PartitionLists.Time: 7 (7)
> - CatalogFetch.Partitions.Hits: 1837 (1837)
> - CatalogFetch.Partitions.Misses: 1837 (1837)
> - CatalogFetch.Partitions.Requests: 3674 (3674)
> - CatalogFetch.Partitions.Time: 325 (325)
> - CatalogFetch.RPCs.Bytes: 4.7 MiB (4936030)
> - CatalogFetch.RPCs.Requests: 22 (22)
> - CatalogFetch.RPCs.Time: 343 (343)
> - CatalogFetch.TableNames.Hits: 4 (4)
> - CatalogFetch.TableNames.Misses: 4 (4)
> - CatalogFetch.TableNames.Requests: 8 (8)
> - CatalogFetch.TableNames.Time: 0 (0)
> - CatalogFetch.Tables.Misses: 8 (8)
> - CatalogFetch.Tables.Requests: 8 (8)
> - CatalogFetch.Tables.Time: 74 (74)
> - InactiveTotalTime: 0ns (0)
> - TotalTime: 0ns (0)
> ImpalaServer
> - CatalogOpExecTimer: 1.97s (1972007962)
> - ClientFetchWaitTimer: 0ns (0)
> - InactiveTotalTime: 0ns (0)
> - RowMaterializationTimer: 0ns (0)
> - TotalTime: 0ns (0)
> Child Queries
> Table Stats Query (id=db4821e4aa5bb04d:d4a5ae4500000000)
> Column Stats Query (id=0444367557e3496d:f943511100000000)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org