Set AM Node Label Expression for Tez Application

[Kushal Khandewal <ku...@hotstar.com>]

Hi,

I am running Hive Queries with Tez execution engine.
Is there a way to run  AM on a particular queue / node_label? I need this
to ensure that Application Master always runs on dedicated nodes.

Best,
Kushal

-- 
This message and its
attachments are confidential (or legally privileged) 
information and are meant
solely for the addressee of such message. Any 
unauthorized use of the message /
its attachments is strictly prohibited.

Re: Set AM Node Label Expression for Tez Application

[Sungwoo Park <gl...@gmail.com>]

To specify a queue, one can set "tez.queue.name" in tez-site.xml.

As far as I know, Tez does not currently support specifying a node label
for Application Master. However, it is pretty easy to implement this
feature by extending
tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java in the Tez
distribution. You could add the following line at the end of
createApplicationSubmissionContext() in TezClientUtils.java and rebuild Tez:

appContext.setNodeLabelExpression("node-label-for-your-application-master");

If you don't mind learning a new execution engine for Hive, MR3 (
https://mr3.postech.ac.kr) supports this. When AMProcess mode is enabled,
Application Master runs as a separate process on the same node where
HiveServer2 runs. So, you could just run HiveServer2 on a node of your
choice (outside the Hadoop cluster). If you want to run Application Master
as a Hadoop container on a node with a specific node label, I can include
this feature in MR3 0.10 (to be released soon). As a matter of fact, it
would be a good idea to try MR3 :-) You can get all the benefit of
Hive-LLAP and much more without the hassle of setting up ZooKeeper, slider,
LLAP daemons, etc. (For high availability, you need ZooKeeper.)

Cheers,

--- Sungwoo



On Fri, Sep 13, 2019 at 8:27 PM Kushal Khandewal <ku...@hotstar.com> wrote:

> Hi,
>
> I am running Hive Queries with Tez execution engine.
> Is there a way to run  AM on a particular queue / node_label? I need this
> to ensure that Application Master always runs on dedicated nodes.
>
> Best,
> Kushal
>
> This message and its attachments are confidential (or legally privileged)
> information and are meant solely for the addressee of such message. Any
> unauthorized use of the message / its attachments is strictly prohibited.