You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org> on 2021/06/03 01:49:09 UTC
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Yu-Wen Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17538
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
IMPALA-10724: Add mutable validWriteIdList
In this patch, we add a new class for manually updating writeIdList.
In terms of updating writeIdList, we introduce three methods including
addOpenWriteId, addAbortedWriteIds, and addCommittedWriteIds.
We will use this class in MetastoreEventProcessor for fine-grained
table refreshing. With the control of writeIdList, we will be able to
update the transactional table partially and keep it consistent.
There are some restrictions for MutableValidWriteIdList.
1. We need to mark a writeId open before mark it committed/aborted.
2. We only allow two writeId state transitions, open -> committed or
open -> aborted. Any other transition is NOT allowed.
Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidWriteIdList.java
A fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
4 files changed, 557 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/17538/1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2701
PS1, Line 2701: public MutableValidWriteIdList getMutableValidWriteIds() {
> Can you not use the previous method itself ?
The previous method is inherited from parent class Table and I shouldn't change the method signature since we only need mutable writeIdList for HdfsTable.
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
File fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java:
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@68
PS1, Line 68: public RangeResponse isWriteIdRangeValid(long minWriteId, long maxWriteId) {
> Please add some unit tests for all the variations of RangeResponse.
Added test in testRangeResponse
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@69
PS1, Line 69: // check the easy cases first
> You can remove this comment
Done
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@72
PS1, Line 72: } else if (exceptions.size() > 0 && exceptions.get(0) > maxWriteId) {
> You don't need "else"
Done
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@87
PS1, Line 87: } else if (count == (maxWriteId - minWriteId + 1)) {
> You can remove the elses
Done
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@214
PS1, Line 214: public RangeResponse isWriteIdRangeAborted(long minWriteId, long maxWriteId) {
> Please add some unit tests for all the variations.
Added tests in method TestRangeResponse
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@235
PS1, Line 235: } else if (count == (maxWriteId - minWriteId + 1)) {
> Remove the elses
Done
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 16:42:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 23:13:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8838/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 02:11:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
IMPALA-10724: Add mutable validWriteIdList
In this patch, we add a new class for manually updating writeIdList.
In terms of updating writeIdList, we introduce three methods including
addOpenWriteId, addAbortedWriteIds, and addCommittedWriteIds.
We will use this class in MetastoreEventProcessor for fine-grained
table refreshing. With the control of writeIdList, we will be able to
update the transactional table partially and keep it consistent.
There are some restrictions for MutableValidWriteIdList.
1. We need to mark a writeId open before mark it committed/aborted.
2. We only allow two writeId state transitions, open -> committed or
open -> aborted. Any other transition is NOT allowed.
Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidWriteIdList.java
A fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
4 files changed, 580 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/17538/3
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 22:59:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 6
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Jul 2021 23:45:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 07:49:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
IMPALA-10724: Add mutable validWriteIdList
In this patch, we add a new class for manually updating writeIdList.
In terms of updating writeIdList, we introduce three methods including
addOpenWriteId, addAbortedWriteIds, and addCommittedWriteIds.
We will use this class in MetastoreEventProcessor for fine-grained
table refreshing. With the control of writeIdList, we will be able to
update the transactional table partially and keep it consistent.
There are some restrictions for MutableValidWriteIdList.
1. We need to mark a writeId open before mark it committed/aborted.
2. We only allow two writeId state transitions, open -> committed or
open -> aborted. Any other transition is NOT allowed.
Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Reviewed-on: http://gerrit.cloudera.org:8080/17538
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidWriteIdList.java
A fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
4 files changed, 573 insertions(+), 4 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Quanlong Huang: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 7
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 4: Code-Review+2
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17538/4/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
File fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java:
http://gerrit.cloudera.org:8080/#/c/17538/4/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@112
PS4, Line 112: if (exceptions.size() == 0) {
: buf.append(':');
: buf.append(':');
: }
nit: this is covered by the else-clause so can be removed.
http://gerrit.cloudera.org:8080/#/c/17538/4/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@146
PS4, Line 146: String[] values = src.split(":");
Could you add a precondition check like this?
Preconditions.checkState(values.length >= 3, "Not enough values");
http://gerrit.cloudera.org:8080/#/c/17538/4/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@171
PS4, Line 171: int i = 0;
nit: unused var
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Jul 2021 06:58:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8887/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 17:25:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 7:
LGTM.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 7
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Jul 2021 17:40:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 6:
Pushing this since the GVO passed. Thank Yu-Wen for this work!
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 6
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Jul 2021 00:34:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
IMPALA-10724: Add mutable validWriteIdList
In this patch, we add a new class for manually updating writeIdList.
In terms of updating writeIdList, we introduce three methods including
addOpenWriteId, addAbortedWriteIds, and addCommittedWriteIds.
We will use this class in MetastoreEventProcessor for fine-grained
table refreshing. With the control of writeIdList, we will be able to
update the transactional table partially and keep it consistent.
There are some restrictions for MutableValidWriteIdList.
1. We need to mark a writeId open before mark it committed/aborted.
2. We only allow two writeId state transitions, open -> committed or
open -> aborted. Any other transition is NOT allowed.
Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidWriteIdList.java
A fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
4 files changed, 573 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/17538/5
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 5
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7218/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 17:00:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7191/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 01:52:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 6
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Jul 2021 00:33:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9077/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 5
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Jul 2021 18:05:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8923/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 18:17:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
IMPALA-10724: Add mutable validWriteIdList
In this patch, we add a new class for manually updating writeIdList.
In terms of updating writeIdList, we introduce three methods including
addOpenWriteId, addAbortedWriteIds, and addCommittedWriteIds.
We will use this class in MetastoreEventProcessor for fine-grained
table refreshing. With the control of writeIdList, we will be able to
update the transactional table partially and keep it consistent.
There are some restrictions for MutableValidWriteIdList.
1. We need to mark a writeId open before mark it committed/aborted.
2. We only allow two writeId state transitions, open -> committed or
open -> aborted. Any other transition is NOT allowed.
Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidWriteIdList.java
A fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
4 files changed, 576 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/17538/4
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 4: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Jul 2021 18:11:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7219/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 17:06:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2701
PS1, Line 2701: public MutableValidWriteIdList getMutableValidWriteIds() {
> The previous method is inherited from parent class Table and I shouldn't ch
Why do you have to change the method signature ?
MutableValidWriteIdList extends ValidWriteIdList, so you can still the use the same method.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Jun 2021 21:36:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Jul 2021 05:33:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7275/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Jul 2021 23:38:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17538/2/fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
File fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java:
http://gerrit.cloudera.org:8080/#/c/17538/2/fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java@101
PS2, Line 101: assertEquals(ValidWriteIdList.RangeResponse.NONE, mutableWriteIdList.isWriteIdRangeAborted(0, 99));
line too long (103 > 90)
http://gerrit.cloudera.org:8080/#/c/17538/2/fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java@105
PS2, Line 105: assertEquals(ValidWriteIdList.RangeResponse.ALL, mutableWriteIdList.isWriteIdRangeAborted(100, 1099));
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/17538/2/fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java@109
PS2, Line 109: assertEquals(ValidWriteIdList.RangeResponse.SOME, mutableWriteIdList.isWriteIdRangeAborted(100, 2000));
line too long (107 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 16:59:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7288/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 6
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Jul 2021 17:48:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8886/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 17:18:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2701
PS1, Line 2701: public MutableValidWriteIdList getMutableValidWriteIds() {
Can you not use the previous method itself ?
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
File fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java:
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@68
PS1, Line 68: public RangeResponse isWriteIdRangeValid(long minWriteId, long maxWriteId) {
Please add some unit tests for all the variations of RangeResponse.
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@69
PS1, Line 69: // check the easy cases first
You can remove this comment
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@72
PS1, Line 72: } else if (exceptions.size() > 0 && exceptions.get(0) > maxWriteId) {
You don't need "else"
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@87
PS1, Line 87: } else if (count == (maxWriteId - minWriteId + 1)) {
You can remove the elses
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@214
PS1, Line 214: public RangeResponse isWriteIdRangeAborted(long minWriteId, long maxWriteId) {
Please add some unit tests for all the variations.
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@235
PS1, Line 235: } else if (count == (maxWriteId - minWriteId + 1)) {
Remove the elses
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 21:28:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/17538/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2701
PS1, Line 2701: public MutableValidWriteIdList getMutableValidWriteIds() {
> Why do you have to change the method signature ?
Oh I see. We can explicitly cast it to MutableValidWriteIdList when we need to update it. Will remove this.
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Jun 2021 21:53:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10724: Add mutable validWriteIdList
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/17538 )
Change subject: IMPALA-10724: Add mutable validWriteIdList
......................................................................
IMPALA-10724: Add mutable validWriteIdList
In this patch, we add a new class for manually updating writeIdList.
In terms of updating writeIdList, we introduce three methods including
addOpenWriteId, addAbortedWriteIds, and addCommittedWriteIds.
We will use this class in MetastoreEventProcessor for fine-grained
table refreshing. With the control of writeIdList, we will be able to
update the transactional table partially and keep it consistent.
There are some restrictions for MutableValidWriteIdList.
1. We need to mark a writeId open before mark it committed/aborted.
2. We only allow two writeId state transitions, open -> committed or
open -> aborted. Any other transition is NOT allowed.
Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
A fe/src/main/java/org/apache/impala/hive/common/MutableValidWriteIdList.java
A fe/src/test/java/org/apache/impala/hive/common/MutableValidReaderWriteIdListTest.java
4 files changed, 577 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/17538/2
--
To view, visit http://gerrit.cloudera.org:8080/17538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I28e60db0afd5d4398af24449b72abc928421f7c6
Gerrit-Change-Number: 17538
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>