Fetching input

[Rohini Palaniswamy <ro...@gmail.com>]

Hi,
   Does tez always fetch input from previous vertex through NodeManager or
it optimizes by fetching from local directory if task is scheduled on same
container/node?  For eg: if container is reused on a 1-1 edge, it would be
more optimal to read from task local directory than fetching through
NodeManager jetty.

Regards,
Rohini

Re: Fetching input

[Siddharth Seth <ss...@apache.org>]

We're also looking at Tez having it's own NodeManager plugin instead of
relying on MapReduce Shuffle. MR-Shuffle is fairly limited in what it does,
and gets in the way of writing multiple files, formats etc.
Ideally, the Tez plugin would be compatible with MR - so that only one
needs to be run on the NodeManager.


On Wed, Apr 23, 2014 at 12:35 PM, Bikas Saha <bi...@hortonworks.com> wrote:

> That is TBD and also required for the local runner case. The Fetcher should
> be able to read from local disk directly if it can.
>
> -----Original Message-----
> From: Rohini Palaniswamy [mailto:rohini.aditya@gmail.com]
> Sent: Wednesday, April 23, 2014 12:18 PM
> To: dev@tez.incubator.apache.org
> Subject: Fetching input
>
> Hi,
>    Does tez always fetch input from previous vertex through NodeManager or
> it optimizes by fetching from local directory if task is scheduled on same
> container/node?  For eg: if container is reused on a 1-1 edge, it would be
> more optimal to read from task local directory than fetching through
> NodeManager jetty.
>
> Regards,
> Rohini
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

RE: Fetching input

[Bikas Saha <bi...@hortonworks.com>]

That is TBD and also required for the local runner case. The Fetcher should
be able to read from local disk directly if it can.

-----Original Message-----
From: Rohini Palaniswamy [mailto:rohini.aditya@gmail.com]
Sent: Wednesday, April 23, 2014 12:18 PM
To: dev@tez.incubator.apache.org
Subject: Fetching input

Hi,
   Does tez always fetch input from previous vertex through NodeManager or
it optimizes by fetching from local directory if task is scheduled on same
container/node?  For eg: if container is reused on a 1-1 edge, it would be
more optimal to read from task local directory than fetching through
NodeManager jetty.

Regards,
Rohini

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.