[jira] [Commented] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools

["Peter Bacsko (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224551#comment-17224551 ] 

Peter Bacsko commented on YARN-10458:
-------------------------------------

I uploaded a patch to branch-3.3. Older branches don't contain the existing code (username from apptags), so it's the only branch which is worth backporting to.

> Hive On Tez queries fails upon submission to dynamically created pools
> ----------------------------------------------------------------------
>
>                 Key: YARN-10458
>                 URL: https://issues.apache.org/jira/browse/YARN-10458
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Anand Srinivasan
>            Assignee: Peter Bacsko
>            Priority: Major
>             Fix For: 3.4.0
>
>         Attachments: YARN-10458-001.patch, YARN-10458-002.patch, YARN-10458-003.patch, YARN-10458-004.patch, YARN-10458-branch-3.3.001.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see that the queue creation fails because ACL submit application check couldn't succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues. For static queues, this worked but failed for dynamic queues. Also tried setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule: Application application_1600399068816_0460 user user1 mapping [default] to [queue1] override false
> 2020-09-18 01:08:40,579 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from application tag does not have access to  queue 'user1'. The placement is done for user 'hive'
>  
> Checking the code, scheduler#checkAccess() bails out even before checking the ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " + queueName); }
> return false;                    *<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be scheduler#checkAccess() should have a logic to differentiate when CSQueue is null and if queue mapping is involved and if so, check if the parent queue exists and is a managed parent and if so, check if the parent queue has valid ACL's instead of returning false ?
> Thanks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org