You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "zhangbutao (Jira)" <ji...@apache.org> on 2019/11/27 08:58:00 UTC
[jira] [Assigned] (HIVE-22527) Hive on Tez : Job of merging samll
files will be submitted into another queue (default queue)
[ https://issues.apache.org/jira/browse/HIVE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhangbutao reassigned HIVE-22527:
---------------------------------
Assignee: zhangbutao
> Hive on Tez : Job of merging samll files will be submitted into another queue (default queue)
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-22527
> URL: https://issues.apache.org/jira/browse/HIVE-22527
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.0, 3.1.1
> Reporter: zhangbutao
> Assignee: zhangbutao
> Priority: Blocker
> Attachments: explain with merge files.png, file merge job.png, hive logs.png
>
>
> Hive on Tez. We enable small file merge configuration with set *hive.merge.tezfiles=true*. So , There will be another job launched for merging files after sql job. However, the merge file job is submitted into another yarn queue, not the queue of current beeline client session. It seems that the merging files job start a new tez session with new conf which is different the current session conf, leading to the merging file job goes into default queue.
>
> Attachment *hive logs.png* shows that current session queue is *root.bdoc.production* ( String queueName = session.getQueueName();) incoming queue name is *null* ( String confQueueName = conf.get(TezConfiguration.TEZ_QUEUE_NAME);). In fact, we log in to the same beeline client with *set tez.queue.name=* *root.bdoc.production,* and all jobs should be submitted into the same queue including file merge job.
> [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L445]
> [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L446]
>
> Attachment *explain with merge files.png* shows that ** the stage-4 is individual merge file job which is submitted into another yarn queue（default queue）， not the queue root.bdoc.production.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)