You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/12/21 14:18:01 UTC
[GitHub] [hive] marton-bod opened a new pull request #1797: HIVE-24541: add config option for default storage handler class
marton-bod opened a new pull request #1797:
URL: https://github.com/apache/hive/pull/1797
### What changes were proposed in this pull request?
Add new config option to set a default storage handler class. If this config is set to a valid, existing class, the storage handler class won't have to be explicitly provided in the create table statement with the `STORED BY `clause, but will be 'inserted' under the hood.
For example, if `hive.default.storage.handler.class=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler` is set,
then the command `CREATE TABLE abc (a int, b string) LOCATION ...` will be equivalent to
`CREATE TABLE abc (a int, b string) STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' LOCATION ...`
### Why are the changes needed?
In order to use existing create table commands for new table formats, such as Iceberg, without modification.
### Does this PR introduce _any_ user-facing change?
new config option settable via beeline/hive-site
### How was this patch tested?
q tests + manually with local run
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary merged pull request #1797: HIVE-24541: add config option for default storage handler class
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #1797:
URL: https://github.com/apache/hive/pull/1797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #1797: HIVE-24541: add config option for default storage handler class
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #1797:
URL: https://github.com/apache/hive/pull/1797#discussion_r551356985
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/StorageFormat.java
##########
@@ -69,6 +69,12 @@ public boolean fillStorageFormat(ASTNode child) throws SemanticException {
}
break;
case HiveParser.TOK_FILEFORMAT_GENERIC:
+ if (storageHandler != null) {
+ // Under normal circumstances, we should not get here (since STORED BY and STORED AS are incompatible within the
+ // same command). Only scenario we can end up here is if a default storage handler class is set in the config.
+ // In this case, we opt to ignore the STORED AS clause.
Review comment:
sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1797: HIVE-24541: add config option for default storage handler class
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1797:
URL: https://github.com/apache/hive/pull/1797#discussion_r551330709
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -1991,6 +1991,10 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"Default file format for CREATE TABLE statement applied to managed tables only. External tables will be \n" +
"created with format specified by hive.default.fileformat. Leaving this null will result in using hive.default.fileformat \n" +
"for all tables."),
+ HIVEDEFAULTSTORAGEHANDLER("hive.default.storage.handler.class", "",
Review comment:
Could be please use underscores to separate names? I understand that the ones before and after are not using them, but currently the underscore is the "standard"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1797: HIVE-24541: add config option for default storage handler class
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1797:
URL: https://github.com/apache/hive/pull/1797#discussion_r551331191
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/StorageFormat.java
##########
@@ -69,6 +69,12 @@ public boolean fillStorageFormat(ASTNode child) throws SemanticException {
}
break;
case HiveParser.TOK_FILEFORMAT_GENERIC:
+ if (storageHandler != null) {
+ // Under normal circumstances, we should not get here (since STORED BY and STORED AS are incompatible within the
+ // same command). Only scenario we can end up here is if a default storage handler class is set in the config.
+ // In this case, we opt to ignore the STORED AS clause.
Review comment:
Maybe log?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] marton-bod commented on a change in pull request #1797: HIVE-24541: add config option for default storage handler class
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #1797:
URL: https://github.com/apache/hive/pull/1797#discussion_r551357048
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -1991,6 +1991,10 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"Default file format for CREATE TABLE statement applied to managed tables only. External tables will be \n" +
"created with format specified by hive.default.fileformat. Leaving this null will result in using hive.default.fileformat \n" +
"for all tables."),
+ HIVEDEFAULTSTORAGEHANDLER("hive.default.storage.handler.class", "",
Review comment:
sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org