get input size for each task

[Grandl Robert <rg...@yahoo.com>]

Hi guys,

It is possible to know for a task/vertex what is the input size it needs to transfer from each input task / vertex on every edge ? Similar, or the same for output ? 


I know for each task/vertex you know the input/output vertices, but I could not find a way to determine the input size on each edge to these vertices ? 

Thanks,
Robert

RE: get input size for each task

[Bikas Saha <bi...@hortonworks.com>]

There isn’t a way currently. Each input and output however, have that
information and could potentially publish that. It may be tractable to
publish/store such information per logical edge. Ie. For each task state
how much data its reading in aggregate from each of its input vertices.
However, storing that per input task (instead of per input vertex) would be
O(m*n) for a scatter edge connecting m source and n destination tasks. We
would like to avoid that for memory allocation reasons.



Bikas



*From:* Grandl Robert [mailto:rgrandl@yahoo.com]
*Sent:* Sunday, July 06, 2014 3:23 PM
*To:* user@tez.incubator.apache.org
*Subject:* get input size for each task



Hi guys,



It is possible to know for a task/vertex what is the input size it needs to
transfer from each input task / vertex on every edge ? Similar, or the same
for output ?



I know for each task/vertex you know the input/output vertices, but I could
not find a way to determine the input size on each edge to these vertices ?



Thanks,

Robert

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.