[I] [CPP] Reader misreads every page after an empty value page (IoTDB aligned series with an all-null page) [tsfile]

["fl0-m (via GitHub)" <gi...@apache.org>]

fl0-m opened a new issue, #971:
URL: https://github.com/apache/tsfile/issues/971

   **Version:** apache/tsfile `cpp`, commit 7ca2e79 and `develop` 83fe559 (2026-09-24). Files written by IoTDB 2.0.11 (table model, aligned, LZ4).
   
   **What happens.** IoTDB's Java writer stores a value page in which a column has no value as a single varint `0` (`ValueChunkWriter.writeEmptyPageToPageBuffer`), and the Java reader returns an empty page as soon as it reads that `0` (`PageHeader.deserializeFrom`: `if (uncompressedSize == 0) // Empty Page`). The C++ `PageHeader::deserialize_from` (`cpp/src/common/tsfile_common.h`) always reads a second varint (compressed size) and, when `deserialize_stat`, a statistic. After an empty page it therefore consumes the next page's bytes as this page's header, and every later page of that column is misread.
   
   **Minimal reproduction.** In IoTDB, one table with two INT32 fields `a`, `b`; write 20,000 rows in tablets of 1,000, `b` filled in the 1st, 3rd, 5th … tablet and null in the others; `FLUSH`. Reading `b` (or `a, b`) from the sealed file through the C++ reader (e.g. the DuckDB `tsfile` extension) returns **0 rows**; the truth is 20,000 rows, 10,000 non-null, sum 94,995,000. The same data with `b` filled in the first half and null after, or on every other row, reads correctly (the empty page must be followed by a non-empty one).
   
   **Fix that works for us** (in `PageHeader::deserialize_from`, after reading `uncompressed_size_`):
   ```cpp
   if (uncompressed_size_ == 0) {   // empty page: one varint, no size, no statistics
       compressed_size_ = 0;
       statistic_ = nullptr;
       return common::E_OK;
   }
   ```
   The aligned readers already treat `compressed_size_ == 0` as an all-null page. With it, 574 of 574 affected column reads over 26 real IoTDB files matched the file metadata (0 of 574 before).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@tsfile.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [CPP] Reader misreads every page after an empty value page (IoTDB aligned series with an all-null page) [tsfile]

["ColinLeeo (via GitHub)" <gi...@apache.org>]

ColinLeeo closed issue #971: [CPP] Reader misreads every page after an empty value page (IoTDB aligned series with an all-null page)
URL: https://github.com/apache/tsfile/issues/971


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@tsfile.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [CPP] Reader misreads every page after an empty value page (IoTDB aligned series with an all-null page) [tsfile]

["ColinLeeo (via GitHub)" <gi...@apache.org>]

ColinLeeo commented on issue #971:
URL: https://github.com/apache/tsfile/issues/971#issuecomment-5888689555

   Thanks for reporting this.
    
   We checked the current C++ implementation and confirmed the issue. When uncompressed_size_ is zero, PageHeader::deserialize_from still reads the compressed size and optional statistics, while an empty page written by Java contains only the single zero varint. 
   
   This causes the reader to consume bytes from the following page and misalign subsequent page parsing. The suggested early return is the correct fix. We will address it and add a regression test.
   
   We plan to fully address this issue, together with a regression test, in the upcoming 2.5.0 release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@tsfile.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org