[GitHub] [incubator-uniffle] jerqi commented on issue #239: [Problem] RssUtils#transIndexDataToSegments should consider the length of the data file

[GitBox <gi...@apache.org>]

jerqi commented on issue #239:
URL: https://github.com/apache/incubator-uniffle/issues/239#issuecomment-1286395993

   > > > You are right, we need to know how much data can be read before HdfsFileReader#read, and then read the corresponding data through HdfsFileReader#read, but how much data can be read is currently obtained by parsing the index file , I think the actual amount of data that can be read needs to be considered in the process of parsing the index file, so as to avoid EOFException. Just like the last block in the picture above, when parsing the index file, it is found that there is no such block according to the length of the data file, and this block should not be added to the segment at this time.
   > > 
   > > 
   > > Could we catch the EOFException and return the data directly?
   > 
   > It doesn't seem feasible, because we also need to get the correct bufferSegments, if an EOFEXception is thrown, we don't know which bufferSegments are complete.
   
   I can't get your point. Why don't we know which bufferSegments are complete?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org