[jira] [Comment Edited] (TEZ-2103) Implement a Partial completion VertexManagerPlugin

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380376#comment-14380376 ] 

Hitesh Shah edited comment on TEZ-2103 at 3/25/15 6:06 PM:
-----------------------------------------------------------

Hi [~alokasok], sorry for the late reply. 

The use-case is somewhat along these lines.

Assume that there is a SQL query being run to do something like "select * from table where <filter clauses> limit 10;". This can be broken down into a single stage DAG where each task runs on a partition of data and completes as soon as it has say 10 records that match the filter clauses. 

Now, if the overall data is huge i.e. multiple TBs and needs 100,000 tasks to scan all the data, running 100,000 tasks should not be necessary if the first task to complete returned 10 results. 

VertexManagers are sort of the vertex controllers. They control scheduling of tasks ( when to start running tasks, whether to enable slow start i.e. wait for upstream data to be ready before starting tasks ), locality management, vertex parallelism, etc. The VertexManagerPlugin is a user-provided plugin to modify the above behavior.

If you have read up on Tez and take a more detailed look into the VertexManager code, the VertexManager could be made a bit more powerful to be able to trigger completion of a vertex sooner without needing to run all the tasks if certain conditions get matched earlier. The changes will likely also encompass the Vertex(Impl) state machine in terms of how you treat short-circuited tasks so that all the correct bookkeeping is done from a state management point of view.





was (Author: hitesh):
Hi [~alokasok], sorry for the late reply. 

The use-case is somewhat along these lines.

Assume that there is a SQL query being run to do something like "select * from table where <filter clauses> limit 10;". This can be broken down into a single stage DAG where each task runs on a partition of data and completes as soon as it has say 10 records that match the filter clauses. 

Now, if the overall data is huge i.e. multiple TBs and needs 100,000 tasks to scan all the data, running 100,000 tasks should not be necessary if the first task to complete returned 10 results. 

VertexManagers are sort of the vertex controllers. They control scheduling of tasks ( when to start running tasks, whether to enable slow start i.e. wait for upstream data to be ready before starting tasks ), locality management, vertex parallelism. The VertexManagerPlugin is a user-provided plugin to modify the above behavior.

If you have read up on Tez and take a more detailed look into the VertexManager code, the VertexManager could be made a bit more powerful to be able to trigger completion of a vertex sooner without needing to run all the tasks if certain conditions get matched earlier. The changes will likely also encompass the Vertex(Impl) state machine in terms of how you treat short-circuited tasks so that all the correct bookkeeping is done from a state management point of view.




> Implement a Partial completion VertexManagerPlugin
> --------------------------------------------------
>
>                 Key: TEZ-2103
>                 URL: https://issues.apache.org/jira/browse/TEZ-2103
>             Project: Apache Tez
>          Issue Type: New Feature
>            Reporter: Gopal V
>              Labels: gsoc, gsoc2015, hadoop, java, tez
>
> Currently, there is no sibling communication between tasks - this implies that a task can be completed by the first vertex in a wave of tasks, but the entire wave of tasks has to complete before success can be reported.
> This occurs in limit + filter query patterns common between the data access engines.
> {code}
> select * from data where x > 1 limit 10;
> {code}
> will run through a full-table scan worth of tasks to generate 10 rows per task, to aggregate it to produce the final 10 row result.
> The VertexManager receives counters/events early enough to short-circuit the rest of the vertex tasks, to prevent the remainder of tasks from getting scheduled when the limit condition has been satisfied by an initial sub-set of the tasks.
> This is a specialization of the VertexManagerPlugin for this common case scheduling pattern.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)