[GitHub] [flink] zhijiangW commented on issue #8242: [FLINK-6227][network] Introduce the DataConsumptionException for downstream task failure

[GitBox <gi...@apache.org>]

zhijiangW commented on issue #8242: [FLINK-6227][network] Introduce the DataConsumptionException for downstream task failure
URL: https://github.com/apache/flink/pull/8242#issuecomment-485687507
 
 
   Some thoughts in implementation:
   
   1. The `DataConsumptionException` should be general for both streaming and batch jobs, although we always restart the whole in pipelined mode atm no matter with this `DataConsumptionException`. 
   
   2. We decide to wrap this exception on receiver side, so the sender is not aware of it. The advantages are that it could cover all the cases and reflect the semantics from the aspect of consumer side. This exception is not only involved in the process of `ResultSubpartitionView#getNextBuffer` on sender side, but also involved in the processes of network connection, partition request and data transport.
   
   3. This exception is just a hint for `JobMaster` to decide whether to restart the upstream executions. In future `JobMaster` could also query the partition state from `ShuffleMaster`, then it could consider these two overall factors to make the final decision.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services