[jira] [Commented] (IMPALA-12256) Stale DROP_PARTITION events might not be skipped correctly

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742204#comment-17742204 ] 

ASF subversion and git services commented on IMPALA-12256:
----------------------------------------------------------

Commit fed883379383b368bbce4f3b36b95e01a7378578 in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fed883379 ]

IMPALA-12256: Fix DDLs losing create event ids in reloading partitions

When HMS event-processor is enabled, each partition will track the
event id of the ADD_PARTITION event that corresponds to its creation.
The create event id is used to skip stale DROP_PARTITION events that
have lower event ids.

However, the partition-level create event ids are not correctly retained
during DDLs. DDLs that modify the partition metadata will reload the
partitions. Since HdfsPartition objects are immutable, the reload is
performed as replacing the old one with the new object created by
HdfsPartition.Builder. The HdfsPartition.Builder is built from the old
HdfsPartition object. But it doesn't copy the create event id. This
patch fixes the issue, also adds logs for comparing partition event ids.

Tests:
 - Add e2e tests to verify DDLs won't loss the partition-level create
   event ids, so stale ADD_PARTITION events can be skipped as expected.

Change-Id: I052faa093bda69fb16db0d424c1478bba103dad9
Reviewed-on: http://gerrit.cloudera.org:8080/20145
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Stale DROP_PARTITION events might not be skipped correctly
> ----------------------------------------------------------
>
>                 Key: IMPALA-12256
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12256
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>    Affects Versions: Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, Impala 4.1.2
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>
> Since IMPALA-10502, we track the create event id for db/table/partitions when they are created. It's used to skip stale DROP events, i.e. events that are generated earlier than the object is created.
> However, in some DDLs like COMPUTE INCREMENTAL STATS, we lost the create event id when reloading partitions. This results in stale DROP_PARTITION events not be skipped correctly.
> This can be reproduced with a higher value of "hms_event_polling_interval_s" so the DROP_PARTITION event can come later than the COMPUTE INCREMENTAL STATS finishes.
> {code:bash}
> bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=10 {code}
> Create a non-transactional partitioned table with one partition:
> {code:sql}
> create table my_part (id int) partitioned by (p int) stored as textfile;
> insert into my_part partition(p=0) values (0);{code}
> Put the below commands in a file and run them at once:
> {code:sql}
> alter table my_part drop if exists partition (p=0);
> insert into my_part partition(p=0) values (0),(1),(2),(3);
> compute incremental stats my_part partition(p=0);
> {code}
> In the catalogd logs, we can see the partition being dropped by the DROP_PARTITION event:
> {code:java}
> I0630 13:27:11.840737 17106 CatalogOpExecutor.java:4484] EventId: 8316831 Skipping removal of 0/1 partitions since they don't exist or were created later in table default.my_part.
> I0630 13:27:11.841095 17106 MetastoreEvents.java:628] EventId: 8316831 EventType: DROP_PARTITION 1 partitions dropped from table default.my_part
> {code}
> This event should be skipped since the partition is recreated after it. Although there is a follow-up ADD_PARTITION event (generated by the recreation statement) that will add back the partition, there is a period between them that the metadata is incorrect (missing the actually existing partition).
> The cause is we lost the create_event_id of the recreated partition when reloading it for the COMPUTE INCREMENTAL STATS. There are other DDLs that could cause the same issue, e.g. ALTER TABLE DROP STATS.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org