[jira] [Created] (TEZ-3996) Reorder input failed events before data movement events

["Hitesh Sharma (JIRA)" <ji...@apache.org>]

Hitesh Sharma created TEZ-3996:
----------------------------------

             Summary: Reorder input failed events before data movement events
                 Key: TEZ-3996
                 URL: https://issues.apache.org/jira/browse/TEZ-3996
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Hitesh Sharma


We have a custom processor (AbstractLogicalIOProcessor) that waits for DataMovementEvent to arrive and then starts an external process to do some work. When a revocation happens then the processor recieves an InputFailedEvent, which tells it about the failed input, and we fail the processor as it is working on old inputs. When the new inputs are available then Tez restarts the processor and sends the InputFailedEvent along with all the DataMovementEvent which includes the older versions and the new version that was revocated.

The issue we are seeing is that the events arrive out of order i.e. many times we see the older DataMovementEvent first at which our processor thinks it is good to start. We then receive the InputFailedEvent and the new version of DataMovementEvent, but that's late and the processor fails. This keeps repeating on every subsequent task attempt and the task fails.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)