[jira] [Updated] (HIVE-23802) “merge files” job was submited to default queue when set hive.merge.tezfiles to true

["gaozhan ding (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

gaozhan ding updated HIVE-23802:
--------------------------------
    Attachment: 企业微信截图_15940042679272.png

> “merge files” job was submited to default queue when set hive.merge.tezfiles to true
> ------------------------------------------------------------------------------------
>
>                 Key: HIVE-23802
>                 URL: https://issues.apache.org/jira/browse/HIVE-23802
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 3.1.0
>            Reporter: gaozhan ding
>            Assignee: gaozhan ding
>            Priority: Major
>         Attachments: 企业微信截图_15940042679272.png
>
>
> We use tez as the query engine. When hive.merge.tezfiles  set to true，merge files task,  which followed by orginal task,  will be submit to default queue rather then the queue same with orginal task.
> I study this issue for days and found that, every time starting a container, "tez,queue.name" whill be unset in current session. Code are as below:
> {panel:title=TezSessionState.startSessionAndContainers()}
> // sessionState.getQueueName() comes from cluster wide configured queue names.
> // sessionState.getConf().get("tez.queue.name") is explicitly set by user in a session.
> // TezSessionPoolManager sets tez.queue.name if user has specified one or use the one from
> // cluster wide queue names.
> // There is no way to differentiate how this was set (user vs system).
> // Unset this after opening the session so that reopening of session uses the correct queue
> // names i.e, if client has not died and if the user has explicitly set a queue name
> // then reopened session will use user specified queue name else default cluster queue names.
> conf.unset(TezConfiguration.TEZ_QUEUE_NAME);{panel}
> So after the orgin tast was submited to yarn, "tez,queue.name" will be unset. While starting merge file task, it will try use the same session with orgin job, but get false due to tez,queue.name was unset.
> {panel:title=TezSessionPoolManager.canWorkWithSameSession()}
> if (!session.isDefault()) {
>  String queueName = session.getQueueName();
>  String confQueueName = conf.get(TezConfiguration.TEZ_QUEUE_NAME);
>  LOG.info("Current queue name is " + queueName + " incoming queue name is " + confQueueName);
>  return (queueName == null) ? confQueueName == null : queueName.equals(confQueueName);
> } else {
>  // this session should never be a default session unless something has messed up.
>  throw new HiveException("The pool session " + session + " should have been returned to the pool"); 
> }{panel}
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)