You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (Jira)" <ji...@apache.org> on 2020/06/08 12:05:00 UTC
[jira] [Commented] (HIVE-22036) HMS should identify events
corresponding to replicated database for Atlas HMS hook
[ https://issues.apache.org/jira/browse/HIVE-22036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128233#comment-17128233 ]
Hive QA commented on HIVE-22036:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976105/HIVE-22036.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/22775/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22775/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22775/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-06-08 12:04:02.692
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22775/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-06-08 12:04:02.694
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at d5fb71f HIVE-23605: 'Wrong FS' error during _external_tables_info creation when staging location is remote(Pravin Kumar Sinha, reviewed by Aasha Medhi)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at d5fb71f HIVE-23605: 'Wrong FS' error during _external_tables_info creation when staging location is remote(Pravin Kumar Sinha, reviewed by Aasha Medhi)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-06-08 12:04:03.705
+ rm -rf ../yetus_PreCommit-HIVE-Build-22775
+ mkdir ../yetus_PreCommit-HIVE-Build-22775
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22775
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22775/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java: does not exist in index
error: a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplConst.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterDatabaseEvent.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateDatabaseEvent.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropDatabaseEvent.java: does not exist in index
error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropTableEvent.java: does not exist in index
Trying to apply the patch with -p1
Falling back to three-way merge...
Applied patch to 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/ReplMetaStoreEventListenerTestImpl.java' cleanly.
Falling back to three-way merge...
Applied patch to 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestMetaStoreEventListenerInRepl.java' with conflicts.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java:64
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java' cleanly.
error: patch failed: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplConst.java:30
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplConst.java' with conflicts.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:103
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:84
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterDatabaseEvent.java:33
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterDatabaseEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java:32
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateDatabaseEvent.java:28
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateDatabaseEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java:28
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropDatabaseEvent.java:28
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropDatabaseEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropTableEvent.java:29
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropTableEvent.java' cleanly.
Going to apply patch with: git apply -p1
Falling back to three-way merge...
Applied patch to 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/ReplMetaStoreEventListenerTestImpl.java' cleanly.
Falling back to three-way merge...
Applied patch to 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestMetaStoreEventListenerInRepl.java' with conflicts.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java:64
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java' cleanly.
error: patch failed: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplConst.java:30
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplConst.java' with conflicts.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:103
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:84
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterDatabaseEvent.java:33
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterDatabaseEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java:32
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateDatabaseEvent.java:28
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateDatabaseEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java:28
Falling back to three-way merge...
/data/hiveptest/working/scratch/build.patch:772: new blank line at EOF.
+
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropDatabaseEvent.java:28
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropDatabaseEvent.java' cleanly.
error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropTableEvent.java:29
Falling back to three-way merge...
Applied patch to 'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/DropTableEvent.java' cleanly.
U itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestMetaStoreEventListenerInRepl.java
U standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplConst.java
warning: 1 line adds whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22775
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976105 - PreCommit-HIVE-Build
> HMS should identify events corresponding to replicated database for Atlas HMS hook
> ----------------------------------------------------------------------------------
>
> Key: HIVE-22036
> URL: https://issues.apache.org/jira/browse/HIVE-22036
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Ashutosh Bapat
> Assignee: Ashutosh Bapat
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22036.01.patch, HIVE-22036.02.patch, HIVE-22036.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> An HMS Atlas hook allows an Atlas to create/update/delete its metadata based on the corresponding events in HMS. But Atlas replication happens out-side and before the Hive replication. Thus any events generated during replication may change the Atlas data already replicated, thus interfering with Atlas replication. Hence, provide an HMS interface which the hook can use to identify the events caused by Hive replication flow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)