You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org> on 2022/01/28 04:03:40 UTC
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Yu-Wen Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18175
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
IMPALA-11093: Fine grained table refreshing doesn't refresh table file
metadata
If we insert data into an acid partitioned table from Hive, the
generated events will be like open_txn -> alter_partition
-> commit_txn.
Previously we assumed the partition object with the alter_partition
event has write id < current write id. However, that is not a valid
assumption, the partition object is actually the write id allocated
in this transaction. That means in commit_txn event, we will have
a partition with write id equals to the write id of cached partition.
So we need to modify the '<' condition to '<='.
Tests:
Manually testing
Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
2 files changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/18175/1
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39
PS3, Line 39:
> Yes, the test will fail intermittently without fine-grained table refreshin
Thanks for the clarification.
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Feb 2022 20:39:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7791/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Feb 2022 20:52:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10057/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 04:26:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7772/
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Jan 2022 02:28:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7768/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 04:05:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7772/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 21:46:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 22:07:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Feb 2022 03:29:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 10:33:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39
PS3, Line 39:
> @Yu-Wen: Please confirm the following:
Yes, the test will fail intermittently without fine-grained table refreshing. The issue was that we previously refresh file metadata at alter partition event but while alter partition event was processed the transaction might not be committed yet. If it is committed, we could get new file metadata. Otherwise, we would still see stale file metadata. After my patch, we can now refresh file metadata at commit event.
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Feb 2022 18:06:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
IMPALA-11093: Fine grained table refreshing doesn't refresh table file
metadata
If we insert data into an acid partitioned table from Hive, the
generated events will be like open_txn -> alter_partition
-> commit_txn.
Previously we assumed the partition object with the alter_partition
event has write id < current write id. However, that is not a valid
assumption, the partition object is actually the write id allocated
in this transaction. That means in commit_txn event, we will have
a partition with write id equals to the write id of cached partition.
So we need to modify the '<' condition to '<='.
Tests:
After IMPALA-10923, we now refresh file metadata while processing
commit events. Therefore, we can add back the test disabled in
IMPALA-9057.
Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/metadata/test_event_processing.py
2 files changed, 1 insertion(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/18175/3
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
File fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java:
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@255
PS1, Line 255: updateMinOpenWriteId();
> how is this related to this change?
The minOpenWriteId is not used actually, so no harm as of now. I will remove this from the change and refactor this in another patch.
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 21:42:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39
PS3, Line 39:
@Yu-Wen: Please confirm the following:
1. Are we enabling the test because the flakiness got fixed?
2. Does this test fail if we enable it without the fix in this patch?
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Feb 2022 17:48:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
IMPALA-11093: Fine grained table refreshing doesn't refresh table file
metadata
If we insert data into an acid partitioned table from Hive, the
generated events will be like open_txn -> alter_partition
-> commit_txn.
Previously we assumed the partition object with the alter_partition
event has write id < current write id. However, that is not a valid
assumption, the partition object is actually the write id allocated
in this transaction. That means in commit_txn event, we will have
a partition with write id equals to the write id of cached partition.
So we need to modify the '<' condition to '<='.
Tests:
After IMPALA-10923, we now refresh file metadata while processing
commit events. Therefore, we can add back the test disabled in
IMPALA-9057.
Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Reviewed-on: http://gerrit.cloudera.org:8080/18175
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Sourabh Goyal <so...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/metadata/test_event_processing.py
2 files changed, 1 insertion(+), 2 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Sourabh Goyal: Looks good to me, but someone else must approve
Joe McDonnell: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Feb 2022 21:21:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2763
PS1, Line 2763: || hdfsPartition.getWriteId() <= partition.getWriteId())) {
Please add a test for this since the bug was not caught in the existing test suite.
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
File fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java:
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@255
PS1, Line 255: updateMinOpenWriteId();
how is this related to this change?
On a side note, can we always derive minOpenWriteId from exceptions list and abortedBits? If yes, we can skip maintaining minOpenWriteId
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Jan 2022 12:05:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7776/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Jan 2022 09:03:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
......................................................................
Patch Set 3:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7776/
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Jan 2022 15:34:25 +0000
Gerrit-HasComments: No