[GitHub] [flink] tillrohrmann edited a comment on issue #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend

[GitBox <gi...@apache.org>]

tillrohrmann edited a comment on issue #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#issuecomment-549899085
 
 
   Thanks for addressing Stephan's comments concerning the `SkipListValueSerializer` @carp84. I think it's not worth to apply the same changes to the `SkipListKeySerializer` if it comes at the cost of two additional `System.arraycopy` operations.
   
   > 1. We may still need some modifications on `MemorySegment`. This is to some extent conflict with our original intention of making spill-able backend a standalone module to prevent impact on existing components, so I'm a little bit concerned.
   
   Extending the `MemorySegment` with methods/functionality which could be useful at other places as well is not a problem.
   
   > 2. In the current design we map the file segment into memory (in format of `MappedByteBuffer`) and use it as the base of a "Chunk", furthermore splitting it into "Buckets" for finer-grained usage, so everything in-flight is `ByteBuffer`. Replacing `ByteBuffer` with `MemorySegment` requires many interface changes in `CopyOnWriteSkipListStateMap` and `SpaceAllocator`.
   
   I see the problem. I'm wondering whether it would be easier to do it now as we are opening the PRs and can update them bit by bit or after everything has been merged. I fear that the inertia might be a bit higher if everything is in place and a much bigger pile of code would needed to be touched. But I'm also ok to do it later if you say that this is not a problem.
   
   I know that Stephan also wanted to take a look and then we are hopefully good to move forward with this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services