You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Venugopal Reddy K (Jira)" <ji...@apache.org> on 2023/12/15 10:58:00 UTC
[jira] [Comment Edited] (IMPALA-12356) Partition created by INSERT will make the next ALTER_PARTITION event on it always treated as self-event
[ https://issues.apache.org/jira/browse/IMPALA-12356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797138#comment-17797138 ]
Venugopal Reddy K edited comment on IMPALA-12356 at 12/15/23 10:57 AM:
-----------------------------------------------------------------------
[~stigahuang] Agree with with you on Fix #1. Will modify it accordingly.
Have observed 3 other issues during test. Will fix them with separate jira.
1. create partition table(either from impala or hive), need to make sure table is loaded
2. Alter partition from impala
{code:java}
alter table my_part partition(p=0) set location '/tmp2';{code}
{{}} We set serviceId and version number are added to part params, update to hms and add the version to partition's inFlightEvent. So It is treated as self event in event processsing. {color:#172b4d}*Issue here is we are adding the version to table's inFlightEvent as well(It is not correct as we do not receive alter table event from HMS in this case). So table inFlightEvent has a dangling version number.*{color}
[https://github.com/apache/impala/blob/a6de494f24c47fbd679a037341ae0a34b9f696ff/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1303C1-L1305C78]
3. Now, alter partition from hive beeline
{noformat}
alter table my_part partition(p=0) set location '/tmp3';{noformat}
{color:#172b4d} *During this alter partition event processing, we add to partition's inFlightEvent(It is an issue).*{color}
4. Alter partition from hive beeline again
{noformat}
alter table my_part partition(p=0) set location '/tmp4';{noformat}
Now this alter partition event is treated as self event because of step-3 inFlightEvent issue.
Add a new partition to table with
{noformat}
alter table my_part add partition (p=10);{noformat}
{color:#172b4d}{*}we are adding the version to table's inFlightEvent(B{*}{*}ut we do not receive alter table event from HMS in this case).{*} *So table inFlightEvent has a dangling version number.*{color}
{color:#172b4d}[https://github.com/apache/impala/blob/a6de494f24c47fbd679a037341ae0a34b9f696ff/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1140C1-L1145C82] {color}
was (Author: venureddy):
[~stigahuang] Agree with with you on Fix #1. Will modify it accordingly.
Have observed 3 other issues during test. Will fix them with separate jira.
1. create partition table(either from impala or hive), need to make sure table is loaded
2. Alter partition from impala
{code:java}
alter table my_part partition(p=0) set location '/tmp2';{code}
{{}} We set serviceId and version number are added to part params, update to hms and add the version to partition's inFlightEvent. So It is treated as self event in event processsing. {color:#FF0000}*Issue here is we are adding the version to table's inFlightEvent as well(It is not correct as we do not receive alter table event from HMS in this case)*{color}. So table inFlightEvent has a dangling version number.
[https://github.com/apache/impala/blob/a6de494f24c47fbd679a037341ae0a34b9f696ff/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1303C1-L1305C78]
3. Now, alter partition from hive beeline
{noformat}
alter table my_part partition(p=0) set location '/tmp3';{noformat}
{color:#FF0000}*During this alter partition event processing, we add to partition's inFlightEvent(It is an issue).*{color}
4. Alter partition from hive beeline again
{noformat}
alter table my_part partition(p=0) set location '/tmp4';{noformat}
Now this alter partition event is treated as self event because of step-3 inFlightEvent issue.
Add a new partition to table with
{noformat}
alter table my_part add partition (p=10);{noformat}
{color:#FF0000}{*}we are adding the version to table's inFlightEvent(B{*}{*}ut we do not receive alter table event from HMS in this case).{*} *So table inFlightEvent has a dangling version number.*{color}
{color:#172b4d}{color:#172b4d}[https://github.com/apache/impala/blob/a6de494f24c47fbd679a037341ae0a34b9f696ff/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1140C1-L1145C82] {color}{color}
> Partition created by INSERT will make the next ALTER_PARTITION event on it always treated as self-event
> -------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12356
> URL: https://issues.apache.org/jira/browse/IMPALA-12356
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Venugopal Reddy K
> Priority: Critical
> Labels: ramp-up
>
> In Impala, create a partitioned table and create one partition in it using {*}INSERT{*}:
> {noformat}
> create table my_part (i int) partitioned by (p int) stored as parquet;
> insert into my_part partition(p=0) values (0),(1),(2);
> show partitions my_part
> +-------+-------+--------+------+--------------+-------------------+---------+-------------------+---------------------------------------------------+-----------+
> | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | EC Policy |
> +-------+-------+--------+------+--------------+-------------------+---------+-------------------+---------------------------------------------------+-----------+
> | 0 | -1 | 1 | 358B | NOT CACHED | NOT CACHED | PARQUET | false | hdfs://localhost:20500/test-warehouse/my_part/p=0 | NONE |
> | Total | -1 | 1 | 358B | 0B | | | | | |
> +-------+-------+--------+------+--------------+-------------------+---------+-------------------+---------------------------------------------------+-----------+
> {noformat}
> In Hive, describe the partition. We can see parameters of "impala.events.catalogServiceId" and "impala.events.catalogVersion" added by Impala. This is ok.
> {noformat}
> hive> desc formatted my_part partition(p=0);
> +-----------------------------------+----------------------------------------------------+-----------------------------------+
> | col_name | data_type | comment |
> +-----------------------------------+----------------------------------------------------+-----------------------------------+
> | i | int | |
> | | NULL | NULL |
> | # Partition Information | NULL | NULL |
> | # col_name | data_type | comment |
> | p | int | |
> | | NULL | NULL |
> | # Detailed Partition Information | NULL | NULL |
> | Partition Value: | [0] | NULL |
> | Database: | default | NULL |
> | Table: | my_part | NULL |
> | CreateTime: | Wed Aug 09 15:24:50 CST 2023 | NULL |
> | LastAccessTime: | UNKNOWN | NULL |
> | Location: | hdfs://localhost:20500/test-warehouse/my_part/p=0 | NULL |
> | Partition Parameters: | NULL | NULL |
> | | impala.events.catalogServiceId | eab33ebb8a14cfd:8b2bdc12df3568df |
> | | impala.events.catalogVersion | 1882 |
> | | numFiles | 1 |
> | | totalSize | 358 |
> | | transient_lastDdlTime | 1691565890 |
> | | NULL | NULL |
> | # Storage Information | NULL | NULL |
> | SerDe Library: | org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | NULL |
> | InputFormat: | org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat | NULL |
> | OutputFormat: | org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat | NULL |
> | Compressed: | No | NULL |
> | Num Buckets: | 0 | NULL |
> | Bucket Columns: | [] | NULL |
> | Sort Columns: | [] | NULL |
> +-----------------------------------+----------------------------------------------------+-----------------------------------+
> {noformat}
> Now run an ALTER statement on the partition in Hive, e.g. changing the location:
> {code:sql}
> alter table my_part partition(p=0) set location '/tmp';{code}
> Impala will skip the ALTER_PARTITION event since it's considered as a self-event. In catalogd logs:
> {noformat}
> I0809 15:30:19.628449 29844 MetastoreEvents.java:628] EventId: 8351549 EventType: ALTER_PARTITION Incremented events skipped counter to 12
> I0809 15:30:19.628616 29844 MetastoreEvents.java:628] EventId: 8351549 EventType: ALTER_PARTITION Not processing the event as it is a self-event{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org