[GitHub] [flink] tillrohrmann commented on issue #8242: [FLINK-6227][network] Introduce the DataConsumptionException for downstream task failure

[GitBox <gi...@apache.org>]

tillrohrmann commented on issue #8242: [FLINK-6227][network] Introduce the DataConsumptionException for downstream task failure
URL: https://github.com/apache/flink/pull/8242#issuecomment-489657343
 
 
   Hi @zhijiangW, I think we should not restart the producer if the data transfer caused and exception after establishing the connection and looking up the sub partitions. Otherwise we might restart producers even though there has been only a transient network problem. Thus, I'm not sure whether we should react to the `RemoteInputChannel#onError` wrt to restarting producers.
   
   What kind of environment/hardware issues do you have in mind that could cause repeated failures of reading the result data? Ideally, we would eventually realize that we need to reproduce the result data and, hence, restart the producers.
   
   Did I understand you correctly, that the `PartitionNotFoundException` is good enough and, thus, we don't need to introduce a new exception?
   
   My proposal would be the following:
   
   1. Make sure that `PartitionNotFoundException` is always thrown if lookup fails
   2. Add tests for `PartitionNotFoundException`
   3. If there are problems during data transmission which require the restart of producers then let's introduce a new exception, e.g. `DataTransferException` (being subtype of `DataConsumptionException`, `PartitionNotFoundException` could then also be a subtype of the `DataConsumptionException`)
   4. Update the failover strategy to also consider `DataTransferException` for restarting producers (this could also mean that this exception must occur x times before restarting the producer unlike in the `PartitionNotFoundException` case where it happens immediately)
   
   When you update the PR, please also rebase onto the latest master.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services