[GitHub] [flink] zhijiangW edited a comment on issue #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint

[GitBox <gi...@apache.org>]

zhijiangW edited a comment on issue #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11687#issuecomment-612652864
 
 
   Thanks for the review and suggestions @pnowojski !
   
   Regarding the issue of which thread handling read state, I also fully considered your concerns of netty thread before. Let me further elaborate a bit:
   
   > Netty pool size is configured for handling network traffic, not for reading data from the spilled files. They > should be configured separately, as 32 threads for netty is not that uncommon, while that many threads > for File IO doesn't make much sense.
   
   The default number of netty thread is bounded to slot amount if not config explicitly. I also mentioned about the consideration of how many threads are proper for serving state read in previous comment https://github.com/apache/flink/pull/11687#issuecomment-611465439. Firstly I think one thread should probably be not enough is practice, the reason explained here https://github.com/apache/flink/pull/11687#discussion_r407222896. IMO 10 threads might be an ideally amount in SATA disks, and for SDD we can have more threads concurrent reading in general. Actually the proper size is also related to the current IO pressure, so it is hard to determine beforehand, then I am willing to leave the choice to exiting netty threads to avoid bringing additional thread cost.
   
   It might seem a bit tricky to make netty thread not only work for network traffic by design. But it already exists the same scenarios now that netty thread would read spilled file for blocking partition in batch jobs, even the case is almost the same with reading channel state to some extent. From another aspect, the netty threads on consumer side take the role of feeding data for task processing either from network or from state files. If we do not use netty threads, they are almost idle during state recovery, except receiving new checkpoint barrier occasionally for future supporting. 
   
   > Reading File IO is blocking operation. Blocking Netty threads can cause problems. 
   
   In general I am always against blocking operation in netty thread. But for this case the netty threads should be almost idle during recovery as I mentioned above. And I also considered to support receiving new checkpoint barrier during recovery future, which makes netty thread also work for network read/write ops by internal (e.g. after reading every state buffer). If it takes several minutes to read one buffer in extremely environment to delay new checkpoint receive, I think it should be a minor issue. Because in such bad IO situation, the total recovery might take hours to finish, then supporting minutes level progress for new unaligned checkpoint in this period should be enough. In other words, it seems meaningless if we still support second level progress for new checkpoint during recovery, because the state consumption probably has not been moved forward yet.
   
   In general I like your ideas of the possibility to implement the input recovery from another layer, which I have not thought of before. I agree that it might be better to not touch the existing `RemoteInputChannel` if possible, because it was already involved in many things. I would further think though how many efforts we need to pay if make it work on separate `StreamTaskSpilledInput `/`SpilledInputGate`/`SpilledInputChannel` level or whether it is feasible to work in cleaner way, and feedback to you later.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services