[GitHub] [incubator-iotdb] qiaojialin commented on issue #472: [IOTDB-267]reduce IO operations in deserializing chunk header

[GitBox <gi...@apache.org>]

qiaojialin commented on issue #472: [IOTDB-267]reduce IO operations in deserializing chunk header
URL: https://github.com/apache/incubator-iotdb/pull/472#issuecomment-546269864
 
 
   > > > > > One more push: If we record the total chunk size in the ChunkMetadata, we do not have to split the read of ChunkHeader and chunk data into 2 reads.
   > > > > 
   > > > > 
   > > > > I've thought about it, but there are two other things to consider.
   > > > > 
   > > > > 1. This will change the tsfile structure. I think it's best to change the file structure when upgrading a large version.
   > > > > 2. Although the granularity of data read by iotdb is chunk, we have been discussing whether we should change the granularity of data read by iotdb to page, especially in the case of insufficient memory.
   > > > 
   > > > 
   > > > 
   > > > 1. That is okay, this change could be listed in the change list of 0.9.0. Fear not the changes.
   > > > 2. Reading page by page will surely induce more IO, considering current read performance, I do think it is much beneficial. Moreover, you can adaptively choose to cache a page or a chunk depending on your available memory, which has no conflicts with the current code. And, in my opinion,  trade one IO for only another 8 bytes is so charming.
   > > 
   > > 
   > > I can't get the second point. By `induce` you mean reduce or increase?
   > 
   > Increase, of course. Assuming we have 5 pages in a chunk, you will probably do 6 IOs (plus reading the header) to read the chunk if you read pages one by one.
   
   As far as I know, we do read a Chunk into memory in one IO and then deserialize page one by one in memory.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services