You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Maria <li...@126.com> on 2016/04/24 17:22:08 UTC
How does tez know a map task's outputs route to which reduce task
after auto-reduce done..
Hi,all:
recently I am studying tez shuffle logic. I read "Apache Tez: Dynamic Graph Reconfiguration", and have two questions:
(1)I am not very clear about this "The data samples could be sent via the VertexManager events to the vertex manager that can create the key-range histogram and determine the correct number of partitions. It can then assign the appropriate key-ranges to each partition". How does tez assign the appropriate key-ranges to each partition? by event?
(2)Before auto-reduce, a map task's outputs should go which reduce task is decided by partitioner. But after auto-reduce, number of reduce tasks desc.
how does tez decide that a map task's outputs route to which reduce task after auto-reduce done?
thanks in advance for any reply.
Maria.Lu