You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org> on 2019/07/13 23:10:36 UTC

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13860

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................

IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create
insert events with IllegalStateException.

createInsertEvents() uses partition ids to keep track of the files
added to the partitions by the insert by finding the delta of files
in a partition before and after load() call. However, if partitions
are marked dirty (for eg.: partitions created by RECOVER PARTITIONS),
load() will drop and re-create them which will change the partition
ids. createInsertEvents() then cannot find these parittions and fails
with exception.

In this patch, partitions are tracked by partition names instead of
partition ids so drop + reload will not affect the logic.

Testing:
1. Ran TestRecoverPartitions.test_post_invalidate() which was
failing.
2. Ran MetastoreEventProcessorTest FE tests.

Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
1 file changed, 60 insertions(+), 16 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13860/1
--
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................

IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create
insert events with IllegalStateException.

createInsertEvents() uses partition ids to keep track of the files
added to the partitions by the insert by finding the delta of files
in a partition before and after load() call. However, if partitions
are marked dirty (for eg.: partitions created by RECOVER PARTITIONS),
load() will drop and re-create them which will change the partition
ids. createInsertEvents() then cannot find these parittions and fails
with exception.

In this patch, partitions are tracked by partition names instead of
partition ids so drop + reload will not affect the logic.

Testing:
1. Ran TestRecoverPartitions.test_post_invalidate() which was
   failing.
2. Ran MetastoreEventProcessorTest FE tests.

Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
2 files changed, 35 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13860/4
-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3877/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Jul 2019 23:50:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 18:13:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4659/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 21:14:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 1:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3844
PS1, Line 3844: loadAllPartitions
Is there a reason to load all the partitions? Can we fetch the partitions by name using HDFSTable#getPartitionsForNames(partitionFilesMapBeforeInsert.keySet())


http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3849
PS1, Line 3849: ALTER
Do you mean add partition events?


http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3851
PS1, Line 3851: newPartitionNames
IIUC, this is a list of all the partitionNames which are not affected by this insert operation. Not necessarily, new partitions. Is that right?


http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3855
PS1, Line 3855: partitionNamesMapPostInsert
Its not clear why this is being done. Can you clarify?



-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 18:05:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.

Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 1:

Do you have the full IllegalStateException stack trace? (Don't see it in the jira). I'm trying to understand what the problem is here.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 20:39:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3844
PS1, Line 3844: response.result);
> Is there a reason to load all the partitions? Can we fetch the partitions b
I couldn't use it as it was a private method, may be we should make it public, this is a useful method.


http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3849
PS1, Line 3849: 
> Do you mean add partition events?
Yes, I meant add partitions, thanks for catching that.


http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3851
PS1, Line 3851: sSync_ddl()) {
> IIUC, this is a list of all the partitionNames which are not affected by th
They are new partitions + partitions not affected by this insert. With the change you suggested, this is omitted.


http://gerrit.cloudera.org:8080/#/c/13860/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3855
PS1, Line 3855: turn response;
> Its not clear why this is being done. Can you clarify?
This is done because I was loading all partitions. After your new suggestion on L3844, this can be omitted.



-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 20:32:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Jul 2019 03:39:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................

IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create
insert events with IllegalStateException.

In this patch, partitions are tracked by partition names instead of
partition ids so drop + reload will not affect the logic.

Testing:
1. Ran TestRecoverPartitions.test_post_invalidate() which was
failing.
2. Ran MetastoreEventProcessorTest FE tests.

Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Reviewed-on: http://gerrit.cloudera.org:8080/13860
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
2 files changed, 36 insertions(+), 16 deletions(-)

Approvals:
Impala Public Jenkins: Verified
Vihang Karajgaonkar: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................

IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create
insert events with IllegalStateException.

createInsertEvents() uses partition ids to keep track of the files
added to the partitions by the insert by finding the delta of files
in a partition before and after load() call. However, if partitions
are marked dirty (for eg.: partitions created by RECOVER PARTITIONS),
load() will drop and re-create them which will change the partition
ids. createInsertEvents() then cannot find these parittions and fails
with exception.

In this patch, partitions are tracked by partition names instead of
partition ids so drop + reload will not affect the logic.

Testing:
1. Ran TestRecoverPartitions.test_post_invalidate() which was
   failing.
2. Ran MetastoreEventProcessorTest FE tests.

Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
2 files changed, 51 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13860/2
-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 18:14:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 4:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3868
PS2, Line 3868: tEvents(Table table,
> I think it is more readable if you rename this to partitionsBeforeInsert or
I would keep this name as this name would highlight that the insert events are created only for affected existing partitions.


http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3936
PS2, Line 3936:    */
> static
Done


http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3939
PS2, Line 3939:         return partitions.stream().map(p -> (HdfsPartition) p).collect(
> you could simply this (and below) with stream() and collect()
Done


http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3948
PS2, Line 3948:    * - The table is missing (not yet loaded).
> static?
done



-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 03:16:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2:

@Bharath, I've updated JIRA with repro and some logs.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 21:01:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4647/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 18:14:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2:

(1 comment)

Latest patch looks good to me. I think since this bug can be reproduced reliably, it would be great we should add a test to existing python test for inserts.

http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3868
PS2, Line 3868: affectedExistingPartitions
I think it is more readable if you rename this to partitionsBeforeInsert or partsBeforeInsert



-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2019 21:43:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3911/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 21:12:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/4004/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 21:52:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4647/


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 00:48:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................

IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create
insert events with IllegalStateException.

In this patch, partitions are tracked by partition names instead of
partition ids so drop + reload will not affect the logic.

Testing:
1. Ran TestRecoverPartitions.test_post_invalidate() which was
failing.
2. Ran MetastoreEventProcessorTest FE tests.

Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
2 files changed, 36 insertions(+), 16 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13860/5
--
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3992/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 03:57:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has removed Todd Lipcon from this change.  ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Removed reviewer Todd Lipcon.
-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Jul 2019 22:36:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2:

> (1 comment)
 > 
 > Latest patch looks good to me. I think since this bug can be
 > reproduced reliably, it would be great we should add a test to
 > existing python test for inserts.

Discussed this with Anurag. It seems like there is already a test which fails (TestRecoverPartitions.test_post_invalidate()) due to this bug. Even if we add a new test it will be doing exactly the same as the failing test. Since we have coverage already, we can skip adding a new test.


-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 18:13:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.

Anurag Mantripragada has removed Bharath Vissapragada from this change.  ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Removed reviewer Bharath Vissapragada.
-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.

Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 )

Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
......................................................................


Patch Set 2:

(3 comments)

I won't be able to review this, I'll let Vihang +2 it.

http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3936
PS2, Line 3936:   private Map<String, FeFsPartition> getPartitionNamesMap(Collection<?
static


http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3939
PS2, Line 3939:     for (FeFsPartition part: partitions) {
you could simply this (and below) with stream() and collect()


http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3948
PS2, Line 3948:   private Map<String, Set<String>> getPartitionNameFilesMap(Collection<?
static?



-- 
To view, visit http://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2019 16:44:20 +0000
Gerrit-HasComments: Yes