You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org> on 2020/09/11 01:52:45 UTC
[Impala-ASF-CR] IMPALA-9664: [WIP] Insert event improvements
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16439
Change subject: IMPALA-9664: [WIP] Insert event improvements
......................................................................
IMPALA-9664: [WIP] Insert event improvements
1. Fires insert events for insert overwrite
2. Has file names in the events.
3. Fires transactional notification API.
Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
---
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M tests/custom_cluster/test_event_processing.py
10 files changed, 573 insertions(+), 147 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/16439/1
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7242/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 20:32:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2020 22:38:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16439
to look at the new patch set (#2).
Change subject: IMPALA-9664: Support hive replication
......................................................................
IMPALA-9664: Support hive replication
This patch makes some improvements to the INSERT
event generated by Impala. Specifically, the INSERT event
will now include new file information when Impala inserts
into a table. This information can be used by external
tools like Hive Replication to replicate the changes made
by Impala in their source databases.
Additionally, this patch modifies the truncate table
execution so that it uses HMS API to truncate the table
instead of deleting the files directly on the filesystem.
Following changes were made.
1. Fires insert events for insert overwrite.
2. Has the names of the new files in the events. In case of insert
overwrite, this is just a list of files which were added by
the insert overwrite operation.
3. In case of ACID tables, fires transactional notification API
for all the partitions in which data is inserted.
4. For tables which have replication enabled, the truncate table
operation now uses a HMS API to truncate the table. This is
necessary since HMS API moves the files to a replication change
manager location if needed. Additionally, it generates
ALTER_TABLE events with truncate flag set to true.
TODO:
1. For external tables, replication does not seem to work in the
dev environment. This will be done as a followup.
Testing:
1. Created a new test in test_events_processing.py which inserts
into managed tables which are being replicated. It makes sure that
hive replication detects the new rows which are added into the
tables. The test also exercises insert overwrite and truncate
statements and makes sure that the table is replicated correctly.
Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
---
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/HiveStorageDescriptorFactory.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_event_processing.py
13 files changed, 781 insertions(+), 184 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/16439/2
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/7205/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2020 22:51:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
IMPALA-9664: Support hive replication
This patch makes some improvements to the INSERT
event generated by Impala. Specifically, the INSERT event
will now include new file information when Impala inserts
into a table. This information can be used by external
tools like Hive Replication to replicate the changes made
by Impala in their source databases.
Additionally, this patch modifies the truncate table
execution so that it uses HMS API to truncate the table
instead of deleting the files directly on the filesystem.
Following changes were made.
1. Fires insert events for insert overwrite.
2. Has the names of the new files in the events. In case of insert
overwrite, this is just a list of files which were added by
the insert overwrite operation.
3. In case of ACID tables, fires transactional notification API
for all the partitions in which data is inserted.
4. For tables which have replication enabled, the truncate table
operation now uses a HMS API to truncate the table. This is
necessary since HMS API moves the files to a replication change
manager location if needed. Additionally, it generates
ALTER_TABLE events with truncate flag set to true.
TODO:
1. For external tables, replication does not seem to work in the
dev environment. This will be done as a followup.
Testing:
1. Created a new test in test_events_processing.py which inserts
into managed tables which are being replicated. It makes sure that
hive replication detects the new rows which are added into the
tables. The test also exercises insert overwrite and truncate
statements and makes sure that the table is replicated correctly.
Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Reviewed-on: http://gerrit.cloudera.org:8080/16439
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/HiveStorageDescriptorFactory.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_event_processing.py
13 files changed, 789 insertions(+), 186 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Vihang Karajgaonkar: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 8
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6477/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2020 17:18:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6465/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 20:12:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16439/4/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/16439/4/tests/custom_cluster/test_event_processing.py@363
PS4, Line 363: e
flake8: E722 do not use bare except'
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 20:12:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7301/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2020 17:38:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: [WIP] Insert event improvements
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: [WIP] Insert event improvements
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7157/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Sep 2020 02:13:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16439/6/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/16439/6/tests/custom_cluster/test_event_processing.py@360
PS6, Line 360: e
flake8: E722 do not use bare except'
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2020 17:18:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 7: Code-Review+2
Carrying forward Zoltan's +2 from earlier.
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2020 22:39:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
IMPALA-9664: Support hive replication
This patch makes some improvements to the INSERT
event generated by Impala. Specifically, the INSERT event
will now include new file information when Impala inserts
into a table. This information can be used by external
tools like Hive Replication to replicate the changes made
by Impala in their source databases.
Additionally, this patch modifies the truncate table
execution so that it uses HMS API to truncate the table
instead of deleting the files directly on the filesystem.
Following changes were made.
1. Fires insert events for insert overwrite.
2. Has the names of the new files in the events. In case of insert
overwrite, this is just a list of files which were added by
the insert overwrite operation.
3. In case of ACID tables, fires transactional notification API
for all the partitions in which data is inserted.
4. For tables which have replication enabled, the truncate table
operation now uses a HMS API to truncate the table. This is
necessary since HMS API moves the files to a replication change
manager location if needed. Additionally, it generates
ALTER_TABLE events with truncate flag set to true.
TODO:
1. For external tables, replication does not seem to work in the
dev environment. This will be done as a followup.
Testing:
1. Created a new test in test_events_processing.py which inserts
into managed tables which are being replicated. It makes sure that
hive replication detects the new rows which are added into the
tables. The test also exercises insert overwrite and truncate
statements and makes sure that the table is replicated correctly.
Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
---
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/HiveStorageDescriptorFactory.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_event_processing.py
13 files changed, 789 insertions(+), 186 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/16439/6
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-9664: [WIP] Insert event improvements
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: [WIP] Insert event improvements
......................................................................
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1116
PS1, Line 1116: * @param insertEventDataList A list of insert event encapsulating the information needed
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4598
PS1, Line 4598: makeInsertEventData((HdfsTable) table, partVals, newFiles, isInsertOverwrite));
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@730
PS1, Line 730: List<NotificationEvent> events = ((SynchronousHMSEventProcessorForTests) eventsProcessor_)
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@762
PS1, Line 762: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, false);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@780
PS1, Line 780: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, true);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@885
PS1, Line 885: unpartTable.getFileSystem(), new Path(unpartTable.getHdfsBaseDir()), overwrite, "copy_");
line too long (97 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Sep 2020 01:53:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Sep 2020 13:13:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
IMPALA-9664: Support hive replication
This patch makes some improvements to the INSERT
event generated by Impala. Specifically, the INSERT event
will now include new file information when Impala inserts
into a table. This information can be used by external
tools like Hive Replication to replicate the changes made
by Impala in their source databases.
Additionally, this patch modifies the truncate table
execution so that it uses HMS API to truncate the table
instead of deleting the files directly on the filesystem.
Following changes were made.
1. Fires insert events for insert overwrite.
2. Has the names of the new files in the events. In case of insert
overwrite, this is just a list of files which were added by
the insert overwrite operation.
3. In case of ACID tables, fires transactional notification API
for all the partitions in which data is inserted.
4. For tables which have replication enabled, the truncate table
operation now uses a HMS API to truncate the table. This is
necessary since HMS API moves the files to a replication change
manager location if needed. Additionally, it generates
ALTER_TABLE events with truncate flag set to true.
TODO:
1. For external tables, replication does not seem to work in the
dev environment. This will be done as a followup.
Testing:
1. Created a new test in test_events_processing.py which inserts
into managed tables which are being replicated. It makes sure that
hive replication detects the new rows which are added into the
tables. The test also exercises insert overwrite and truncate
statements and makes sure that the table is replicated correctly.
Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
---
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/HiveStorageDescriptorFactory.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/hive-site.xml.py
M tests/custom_cluster/test_event_processing.py
13 files changed, 792 insertions(+), 186 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/16439/4
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 2:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1131
PS2, Line 1131: * @param insertEventDataList A list of insert event encapsulating the information needed
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4662
PS2, Line 4662: makeInsertEventData((HdfsTable) table, partVals, newFiles, isInsertOverwrite));
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@746
PS2, Line 746: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, false);
> line too long (92 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@764
PS2, Line 764: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, true);
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@869
PS2, Line 869: unpartTable.getFileSystem(), new Path(unpartTable.getHdfsBaseDir()), overwrite, "copy_");
> line too long (97 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@224
PS2, Line 224: _
> flake8: E501 line too long (96 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@226
PS2, Line 226: l
> flake8: E501 line too long (93 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@228
PS2, Line 228: ]
> flake8: E501 line too long (92 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@246
PS2, Line 246: ]
> flake8: E501 line too long (92 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@254
PS2, Line 254: _
> flake8: E501 line too long (96 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@256
PS2, Line 256: l
> flake8: E501 line too long (93 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@265
PS2, Line 265: ]
> flake8: E501 line too long (92 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@274
PS2, Line 274: _
> flake8: E501 line too long (96 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@276
PS2, Line 276: l
> flake8: E501 line too long (93 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@285
PS2, Line 285: ]
> flake8: E501 line too long (92 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@328
PS2, Line 328: ]
> flake8: E501 line too long (92 > 90 characters)
Done
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Sep 2020 00:27:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 2:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1131
PS2, Line 1131: * @param insertEventDataList A list of insert event encapsulating the information needed
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4662
PS2, Line 4662: makeInsertEventData((HdfsTable) table, partVals, newFiles, isInsertOverwrite));
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@746
PS2, Line 746: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, false);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@764
PS2, Line 764: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, true);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@869
PS2, Line 869: unpartTable.getFileSystem(), new Path(unpartTable.getHdfsBaseDir()), overwrite, "copy_");
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@224
PS2, Line 224: _
flake8: E501 line too long (96 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@226
PS2, Line 226: l
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@228
PS2, Line 228: ]
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@246
PS2, Line 246: ]
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@254
PS2, Line 254: _
flake8: E501 line too long (96 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@256
PS2, Line 256: l
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@265
PS2, Line 265: ]
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@274
PS2, Line 274: _
flake8: E501 line too long (96 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@276
PS2, Line 276: l
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@285
PS2, Line 285: ]
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@328
PS2, Line 328: ]
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@350
PS2, Line 350: e
flake8: E722 do not use bare except'
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2020 22:36:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 )
Change subject: IMPALA-9664: Support hive replication
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6465/
--
To view, visit http://gerrit.cloudera.org:8080/16439
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a
Gerrit-Change-Number: 16439
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Sep 2020 01:36:13 +0000
Gerrit-HasComments: No