You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/09/23 11:45:56 UTC
[GitHub] [hive] klcopp opened a new pull request #1518: Hive 24191: Introduce configurable user to run compaction as
klcopp opened a new pull request #1518:
URL: https://github.com/apache/hive/pull/1518
### What changes were proposed in this pull request?
Add a restricted config.
### Why are the changes needed?
Sometimes we want to run compaction as a specific user.
### Does this PR introduce _any_ user-facing change?
Admin can optionally specify a user that will run compaction and list files in compaction Initiator and Cleaner.
### How was this patch tested?
Unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp commented on a change in pull request #1518: HIVE-24191: Introduce configurable user to run compaction as
Posted by GitBox <gi...@apache.org>.
klcopp commented on a change in pull request #1518:
URL: https://github.com/apache/hive/pull/1518#discussion_r493520605
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -3098,6 +3098,10 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"This is set to compactor from within the query based compactor. This enables the Tez SplitGrouper "
+ "to group splits based on their bucket number, so that all rows from different bucket files "
+ " for the same bucket number can end up in the same bucket file after the compaction."),
+ // MetastoreConf doesn't have a restricted list. Otherwise can be moved there.
+ COMPACTOR_RUN_AS_USER("hive.compactor.run.as.user", "",
+ "Specify the user to run compaction (Worker) and compaction-related file listing (Initiator, Cleaner) as. " +
+ "If empty string, defaults to table/partition directory owner. Restricted."),
Review comment:
MetastoreConf, but it has no restricted list AFAIK. Implementing one would be a new feature that isn't covered by this ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp commented on pull request #1518: HIVE-24191: Introduce configurable user to run compaction as
Posted by GitBox <gi...@apache.org>.
klcopp commented on pull request #1518:
URL: https://github.com/apache/hive/pull/1518#issuecomment-697339897
As discussed with @pvary offline: Restricting isn't needed because the Initiator/Worker/Cleaner configs can't be changed without restarting HMS (or HS2 if applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp merged pull request #1518: HIVE-24191: Introduce configurable user to run compaction as
Posted by GitBox <gi...@apache.org>.
klcopp merged pull request #1518:
URL: https://github.com/apache/hive/pull/1518
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1518: HIVE-24191: Introduce configurable user to run compaction as
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1518:
URL: https://github.com/apache/hive/pull/1518#discussion_r493516472
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -3098,6 +3098,10 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"This is set to compactor from within the query based compactor. This enables the Tez SplitGrouper "
+ "to group splits based on their bucket number, so that all rows from different bucket files "
+ " for the same bucket number can end up in the same bucket file after the compaction."),
+ // MetastoreConf doesn't have a restricted list. Otherwise can be moved there.
+ COMPACTOR_RUN_AS_USER("hive.compactor.run.as.user", "",
+ "Specify the user to run compaction (Worker) and compaction-related file listing (Initiator, Cleaner) as. " +
+ "If empty string, defaults to table/partition directory owner. Restricted."),
Review comment:
This will be used on HMS and occasionally on HS2 (if the Worker is HS2).
Where are those other Compaction related configs? If everything is on HMS side, please move this there too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1518: HIVE-24191: Introduce configurable user to run compaction as
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1518:
URL: https://github.com/apache/hive/pull/1518#discussion_r493523854
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -3098,6 +3098,10 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"This is set to compactor from within the query based compactor. This enables the Tez SplitGrouper "
+ "to group splits based on their bucket number, so that all rows from different bucket files "
+ " for the same bucket number can end up in the same bucket file after the compaction."),
+ // MetastoreConf doesn't have a restricted list. Otherwise can be moved there.
+ COMPACTOR_RUN_AS_USER("hive.compactor.run.as.user", "",
+ "Specify the user to run compaction (Worker) and compaction-related file listing (Initiator, Cleaner) as. " +
+ "If empty string, defaults to table/partition directory owner. Restricted."),
Review comment:
AFAIK only very specific HMS confs could be changed by the user (HiveConf.metaConfVars). So restricting it might not be needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp merged pull request #1518: HIVE-24191: Introduce configurable user to run compaction as
Posted by GitBox <gi...@apache.org>.
klcopp merged pull request #1518:
URL: https://github.com/apache/hive/pull/1518
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org