You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/07/14 03:43:40 UTC
[GitHub] [hive] pkumarsinha opened a new pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
pkumarsinha opened a new pull request #1249:
URL: https://github.com/apache/hive/pull/1249
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HIVE-XXXXX: Fix a typo in YYY)
For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1249:
URL: https://github.com/apache/hive/pull/1249#issuecomment-699249395
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1249:
URL: https://github.com/apache/hive/pull/1249#discussion_r456403763
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/CreateFunctionHandler.java
##########
@@ -41,13 +53,36 @@ CreateFunctionMessage eventMessage(String stringRepresentation) {
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} CREATE_FUNCTION message : {}", fromEventId(), eventMessageAsJSON);
Path metadataPath = new Path(withinContext.eventRoot, EximUtil.METADATA_NAME);
+ Path dataPath = new Path(withinContext.eventRoot, EximUtil.DATA_PATH_NAME);
FileSystem fileSystem = metadataPath.getFileSystem(withinContext.hiveConf);
-
+ List<DataCopyPath> functionBinaryCopyPaths = new ArrayList<>();
try (JsonWriter jsonWriter = new JsonWriter(fileSystem, metadataPath)) {
- new FunctionSerializer(eventMessage.getFunctionObj(), withinContext.hiveConf)
- .writeTo(jsonWriter, withinContext.replicationSpec);
+ FunctionSerializer serializer = new FunctionSerializer(eventMessage.getFunctionObj(),
+ dataPath, withinContext.hiveConf);
+ serializer.writeTo(jsonWriter, withinContext.replicationSpec);
+ functionBinaryCopyPaths.addAll(serializer.getFunctionBinaryCopyPaths());
}
withinContext.createDmd(this).write();
+ copyFunctionBinaries(functionBinaryCopyPaths, withinContext.hiveConf);
+ }
+
+ private void copyFunctionBinaries(List<DataCopyPath> functionBinaryCopyPaths, HiveConf hiveConf)
Review comment:
Does this depend on whether copy of load flag is true or false? Or always we will do it at the time of load?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1249:
URL: https://github.com/apache/hive/pull/1249
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1249:
URL: https://github.com/apache/hive/pull/1249#discussion_r455718460
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestReplDumpTask.java
##########
@@ -87,7 +87,9 @@ String getValidTxnListForReplDump(Hive hiveDb, long waitUntilTime) {
}
@Override
- void dumpFunctionMetadata(String dbName, Path dbMetadataRoot, Hive hiveDb) {
+ List<EximUtil.FunctionBinaryCopyPath> dumpFunctionMetadata(String dbName, Path dbMetadataRoot, Path dbDataRoot,
+ Hive hiveDb) {
Review comment:
Not sure but this patch will not change that behavior
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on a change in pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
aasha commented on a change in pull request #1249:
URL: https://github.com/apache/hive/pull/1249#discussion_r454934068
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java
##########
@@ -217,6 +217,52 @@ public static void setNullSrcPath(HiveConf conf, boolean aNullSrcPath) {
}
}
+
+ /**
+ * Wrapper class for mapping source and target path for copying function binaries.
+ */
+ public static class FunctionBinaryCopyPath {
Review comment:
is this class any different from the table copy path. If not can we reuse?
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestReplDumpTask.java
##########
@@ -87,7 +87,9 @@ String getValidTxnListForReplDump(Hive hiveDb, long waitUntilTime) {
}
@Override
- void dumpFunctionMetadata(String dbName, Path dbMetadataRoot, Hive hiveDb) {
+ List<EximUtil.FunctionBinaryCopyPath> dumpFunctionMetadata(String dbName, Path dbMetadataRoot, Path dbDataRoot,
+ Hive hiveDb) {
Review comment:
do we dump function in case of metadata only replication?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aasha commented on pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
aasha commented on pull request #1249:
URL: https://github.com/apache/hive/pull/1249#issuecomment-663357180
Please add a log line for the config value at dump and load time. Will help in debugging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pkumarsinha commented on a change in pull request #1249: HIVE-23835 Repl Dump should dump function binaries to staging directory
Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #1249:
URL: https://github.com/apache/hive/pull/1249#discussion_r456425658
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/CreateFunctionHandler.java
##########
@@ -41,13 +53,36 @@ CreateFunctionMessage eventMessage(String stringRepresentation) {
public void handle(Context withinContext) throws Exception {
LOG.info("Processing#{} CREATE_FUNCTION message : {}", fromEventId(), eventMessageAsJSON);
Path metadataPath = new Path(withinContext.eventRoot, EximUtil.METADATA_NAME);
+ Path dataPath = new Path(withinContext.eventRoot, EximUtil.DATA_PATH_NAME);
FileSystem fileSystem = metadataPath.getFileSystem(withinContext.hiveConf);
-
+ List<DataCopyPath> functionBinaryCopyPaths = new ArrayList<>();
try (JsonWriter jsonWriter = new JsonWriter(fileSystem, metadataPath)) {
- new FunctionSerializer(eventMessage.getFunctionObj(), withinContext.hiveConf)
- .writeTo(jsonWriter, withinContext.replicationSpec);
+ FunctionSerializer serializer = new FunctionSerializer(eventMessage.getFunctionObj(),
+ dataPath, withinContext.hiveConf);
+ serializer.writeTo(jsonWriter, withinContext.replicationSpec);
+ functionBinaryCopyPaths.addAll(serializer.getFunctionBinaryCopyPaths());
}
withinContext.createDmd(this).write();
+ copyFunctionBinaries(functionBinaryCopyPaths, withinContext.hiveConf);
+ }
+
+ private void copyFunctionBinaries(List<DataCopyPath> functionBinaryCopyPaths, HiveConf hiveConf)
Review comment:
no, for function binary copy, we are not using the load flag. It is retained as it is currently. meaning: earlier during load it used to copy from src location. Now with this change, it will copy from staging location. So that src cluster visibility in not required during load of function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org