General question about Tez shuffle

[Andrew Or <an...@gmail.com>]

Hi all,

I have a general question of how Tez shuffle works on a yarn cluster.

I am familiar with how Yarn shuffle uses ShuffleHandler as an auxiliary
service of the node manager to serve spilled files. Does Tez use a similar
interface? I am running an HDP 2.1 cluster with Tez installed, and I
noticed that my "yarn.nodemanager.aux-services" does not include anything
related to Tez. Does Tez run its own HTTP server to serve these files? If
so, are these servers per-node or per-application?

Thanks a lot in advance,
Andrew

RE: General question about Tez shuffle

[Bikas Saha <bi...@hortonworks.com>]

Tez uses the standard yarn shuffle service (per node) that comes packaged
with YARN.

-----Original Message-----
From: Andrew Or [mailto:andrewor14@gmail.com]
Sent: Thursday, September 18, 2014 5:25 PM
To: dev@tez.apache.org
Subject: General question about Tez shuffle

Hi all,

I have a general question of how Tez shuffle works on a yarn cluster.

I am familiar with how Yarn shuffle uses ShuffleHandler as an auxiliary
service of the node manager to serve spilled files. Does Tez use a similar
interface? I am running an HDP 2.1 cluster with Tez installed, and I noticed
that my "yarn.nodemanager.aux-services" does not include anything related to
Tez. Does Tez run its own HTTP server to serve these files? If so, are these
servers per-node or per-application?

Thanks a lot in advance,
Andrew

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.