[jira] [Commented] (TEZ-1447) Handle parallelism updates and versioning w/ custom InputInitializerEvents

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101275#comment-14101275 ] 

Bikas Saha commented on TEZ-1447:
---------------------------------

bq.  Events have a version number, but I don't know which task they are coming from. Thus I can't de-dup events
This can be provided since Tez knows who generated the event.

We need to think of how such notifications can be subscribed to. Perhaps a pub-sub mechanism in the internal state machine to which different entities subscribe and then get notified by the state machine when the state change occurs. 

> Handle parallelism updates and versioning w/ custom InputInitializerEvents
> --------------------------------------------------------------------------
>
>                 Key: TEZ-1447
>                 URL: https://issues.apache.org/jira/browse/TEZ-1447
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.5.0
>            Reporter: Gunther Hagleitner
>            Priority: Blocker
>             Fix For: 0.5.0
>
>
> I'm trying to do dynamic partition pruning through input initializer events in Hive. That means that the initializer of a table scan vertex has to receive events from all tasks in another vertex (which contain the pruning info) before generating tasks to run.
> The problem with the current API I ran into:
> getNumTasks: I'm currently using a busy loop to wait for the num tasks for a vertex to be decided (-1 -> x). There's no way around it, because it's the only way to find out what number of events to expect (0 is a valid number of tasks - so I can't wait for the first to complete).
> With auto-reducer parallelism I have to employ another busy loop. Because I might be initially expecting 10 events, which later get's knocked down to 5. Since there's no event associated with this, I have to periodically check whether I have enough events.
> Versioning: Events have a version number, but I don't know which task they are coming from. Thus I can't de-dup events.



--
This message was sent by Atlassian JIRA
(v6.2#6252)