[GitHub] [flink] curcur edited a comment on pull request #13648: [FLINK-19632] Introduce a new ResultPartitionType for Approximate Local Recovery

[GitBox <gi...@apache.org>]

curcur edited a comment on pull request #13648:
URL: https://github.com/apache/flink/pull/13648#issuecomment-713341581


   Thanks @rkhachatryan so much for the great question on how different threads access the same view (in other words, threading model in netty + task thread interaction on view). Thanks @zhijiangW a lot for the explanation of the underlying threading model based on offline-discussion and great insights.
   
   I've spent some time on different paths of how the view is created/released in different scenarios, here is what I think is happening. @zhijiangW , please correct me if I miss anything, thanks!
   
   For a streaming job, each `view` is created through PartitionRequestServerHandler (one of the netty worker threads)'s PartitionRequest. The created view is owned by subPartition, and a reference of the view is returned and bounded with a reader.  Reader belongs to PartitionRequestServerHandler.
   
   `PipelinedApproximateSubpartition#releaseView` is used in two places:
   1. PipelinedApproximateSubpartition#createView  (From Netty thread, it can see the current version of view belonging to the subpartition)
   2. PipelinedApproximateSubpartition#releaseAllResources,
        This is called from two places, 
        - one from `reader`, the reader uses its own reference of the view to release resources. This is from downstream task cancelation. 
        - one from `subPartition release`. This is fine, the subpartition releases its own view.
   
   Two questions:
   1. Is the old view continue to read data if not disposed of successfully before the new view is created?
   - No, because the reader and the view would be removed up downstream task's cancelation request;
   
   2. What if the netty thread1 release view after netty thread2 recreates the view?
   thread2 release the view thread1 holds the reference before creating a new view.
   When thread1 ties to release the view from the old reference, as long as we disconnects the connection between the old view and the subpartition, we should be fine.
   
   In that sense, we can simply set the view.parent = null when releasing the view.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org