You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@tsfile.apache.org by "hongzhi-gao (via GitHub)" <gi...@apache.org> on 2026/04/30 07:15:03 UTC
[PR] Fix TS_2DIFF float/double page compatibility [tsfile]
hongzhi-gao opened a new pull request, #796:
URL: https://github.com/apache/tsfile/pull/796
## Summary
- Fix C++ TS_2DIFF float/double page layout to match Java overflow-page encoding.
- Harden googletest zip handling in CMake: reject/remove empty or invalid zip files and only accept valid ZIP magic before enabling tests.
## Encoding Layout
**Legacy raw block (old C++ path):**
[ TS_2DIFF inner block ]
**Java-compatible overflow page (fixed C++ path):**
[ outer_magic(varint=2147483646) ]
[ count(varint) ]
[ bitmap_under ]
[ bitmap_over ]
[ inner_max_point(varint=0) ]
[ TS_2DIFF inner block payload ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix TS_2DIFF float/double page compatibility [tsfile]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #796:
URL: https://github.com/apache/tsfile/pull/796#issuecomment-4350564859
## [Codecov](https://app.codecov.io/gh/apache/tsfile/pull/796?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `97.05882%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 62.83%. Comparing base ([`5e2130f`](https://app.codecov.io/gh/apache/tsfile/commit/5e2130f7b715f422edf504ccb9c6380a415aa891?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`247b4ed`](https://app.codecov.io/gh/apache/tsfile/commit/247b4ed4c04314c10f9e32c81c6dea73b0398dc2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/tsfile/pull/796?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [cpp/src/encoding/ts2diff\_decoder.h](https://app.codecov.io/gh/apache/tsfile/pull/796?src=pr&el=tree&filepath=cpp%2Fsrc%2Fencoding%2Fts2diff_decoder.h&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3BwL3NyYy9lbmNvZGluZy90czJkaWZmX2RlY29kZXIuaA==) | 85.71% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/tsfile/pull/796?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #796 +/- ##
===========================================
+ Coverage 62.74% 62.83% +0.09%
===========================================
Files 706 706
Lines 42842 42917 +75
Branches 6326 6354 +28
===========================================
+ Hits 26882 26969 +87
+ Misses 14968 14956 -12
Partials 992 992
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/tsfile/pull/796?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix TS_2DIFF float/double page compatibility [tsfile]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #796:
URL: https://github.com/apache/tsfile/pull/796#discussion_r3347504905
##########
cpp/src/encoding/ts2diff_encoder.h:
##########
@@ -402,5 +484,168 @@ FORCE_INLINE int DoubleTS2DIFFEncoder::encode(double value,
return do_encode(value, out);
}
+// Keep float/double TS_2DIFF page layout compatible with Java.
+FORCE_INLINE int FloatTS2DIFFEncoder::flush(common::ByteStream& out_stream) {
+ int ret = common::E_OK;
+ if (write_index_ == -1) {
+ return common::E_OK;
+ }
+ const int num_values = write_index_ + 1;
+ common::ByteStream inner(1024, common::MOD_TS2DIFF_OBJ, false);
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(max_point_number_), inner))) {
+ return ret;
+ }
+ SIMDOps<int32_t>::rebase(delta_arr_, delta_arr_min_, write_index_);
+ int bit_width = cal_bit_width(delta_arr_max_ - delta_arr_min_);
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(write_index_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(bit_width), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(delta_arr_min_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(first_value_), inner))) {
+ return ret;
+ }
+ for (int i = 0; i < write_index_; i++) {
+ write_bits(delta_arr_[i], bit_width, inner);
+ }
+ flush_remaining(inner);
+ reset();
+
+ const bool overflow = has_overflow();
+ if (overflow) {
+ std::vector<uint8_t> underflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ std::vector<uint8_t> value_overflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ bool has_value_overflow = false;
+ for (int i = 0; i < num_values; i++) {
+ int8_t f = underflow_flags_[static_cast<size_t>(i)];
+ if (f == 1) {
+ underflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ } else if (f == -1) {
+ has_value_overflow = true;
+ value_overflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ }
+ }
+ constexpr uint32_t kJavaOverflowMagic =
+ 2147483647u; // Integer.MAX_VALUE
+ constexpr uint32_t kJavaValueOverflowMagic =
+ 2147483646u; // Integer.MAX_VALUE - 1
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ has_value_overflow ? kJavaValueOverflowMagic
+ : kJavaOverflowMagic,
+ out_stream))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(num_values), out_stream))) {
+ return ret;
+ }
+ const uint32_t bm_len = static_cast<uint32_t>(num_values / 8 + 1);
+ if (RET_FAIL(out_stream.write_buf(underflow_bitmap.data(), bm_len))) {
+ return ret;
+ }
+ if (has_value_overflow && RET_FAIL(out_stream.write_buf(
+ value_overflow_bitmap.data(), bm_len))) {
+ return ret;
+ }
+ }
+ if (RET_FAIL(merge_byte_stream(out_stream, inner, true))) {
+ return ret;
+ }
+ underflow_flags_.clear();
+ return ret;
+}
+
+FORCE_INLINE int DoubleTS2DIFFEncoder::flush(common::ByteStream& out_stream) {
+ int ret = common::E_OK;
+ if (write_index_ == -1) {
+ return common::E_OK;
+ }
+ const int num_values = write_index_ + 1;
+ common::ByteStream inner(1024, common::MOD_TS2DIFF_OBJ, false);
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(max_point_number_), inner))) {
+ return ret;
+ }
+ SIMDOps<int64_t>::rebase(delta_arr_, delta_arr_min_, write_index_);
+ int bit_width = cal_bit_width(delta_arr_max_ - delta_arr_min_);
+ if (RET_FAIL(common::SerializationUtil::write_i32(write_index_, inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_i32(bit_width, inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_i64(delta_arr_min_, inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_i64(first_value_, inner))) {
+ return ret;
+ }
+ for (int i = 0; i < write_index_; i++) {
+ write_bits(delta_arr_[i], bit_width, inner);
+ }
+ flush_remaining(inner);
+ reset();
+
+ const bool overflow = has_overflow();
+ if (overflow) {
+ std::vector<uint8_t> underflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ std::vector<uint8_t> value_overflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
Review Comment:
May use overflow/underflow consistently.
##########
cpp/src/encoding/ts2diff_encoder.h:
##########
@@ -402,5 +484,168 @@ FORCE_INLINE int DoubleTS2DIFFEncoder::encode(double value,
return do_encode(value, out);
}
+// Keep float/double TS_2DIFF page layout compatible with Java.
+FORCE_INLINE int FloatTS2DIFFEncoder::flush(common::ByteStream& out_stream) {
+ int ret = common::E_OK;
+ if (write_index_ == -1) {
+ return common::E_OK;
+ }
+ const int num_values = write_index_ + 1;
+ common::ByteStream inner(1024, common::MOD_TS2DIFF_OBJ, false);
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(max_point_number_), inner))) {
+ return ret;
+ }
+ SIMDOps<int32_t>::rebase(delta_arr_, delta_arr_min_, write_index_);
+ int bit_width = cal_bit_width(delta_arr_max_ - delta_arr_min_);
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(write_index_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(bit_width), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(delta_arr_min_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(first_value_), inner))) {
+ return ret;
+ }
+ for (int i = 0; i < write_index_; i++) {
+ write_bits(delta_arr_[i], bit_width, inner);
+ }
+ flush_remaining(inner);
+ reset();
+
+ const bool overflow = has_overflow();
+ if (overflow) {
+ std::vector<uint8_t> underflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ std::vector<uint8_t> value_overflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ bool has_value_overflow = false;
+ for (int i = 0; i < num_values; i++) {
+ int8_t f = underflow_flags_[static_cast<size_t>(i)];
+ if (f == 1) {
+ underflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ } else if (f == -1) {
+ has_value_overflow = true;
+ value_overflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ }
+ }
+ constexpr uint32_t kJavaOverflowMagic =
+ 2147483647u; // Integer.MAX_VALUE
+ constexpr uint32_t kJavaValueOverflowMagic =
+ 2147483646u; // Integer.MAX_VALUE - 1
Review Comment:
kJavaOverflowMagic -> FLAG_SCALED_VALUE_OVERFLOW
kJavaValueOverflowMagic -> FLAG_ORIGINAL_VALUE_OVERFLOW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix TS_2DIFF float/double page compatibility [tsfile]
Posted by "hongzhi-gao (via GitHub)" <gi...@apache.org>.
hongzhi-gao commented on code in PR #796:
URL: https://github.com/apache/tsfile/pull/796#discussion_r3354524231
##########
cpp/src/encoding/ts2diff_encoder.h:
##########
@@ -402,5 +484,168 @@ FORCE_INLINE int DoubleTS2DIFFEncoder::encode(double value,
return do_encode(value, out);
}
+// Keep float/double TS_2DIFF page layout compatible with Java.
+FORCE_INLINE int FloatTS2DIFFEncoder::flush(common::ByteStream& out_stream) {
+ int ret = common::E_OK;
+ if (write_index_ == -1) {
+ return common::E_OK;
+ }
+ const int num_values = write_index_ + 1;
+ common::ByteStream inner(1024, common::MOD_TS2DIFF_OBJ, false);
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(max_point_number_), inner))) {
+ return ret;
+ }
+ SIMDOps<int32_t>::rebase(delta_arr_, delta_arr_min_, write_index_);
+ int bit_width = cal_bit_width(delta_arr_max_ - delta_arr_min_);
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(write_index_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(bit_width), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(delta_arr_min_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(first_value_), inner))) {
+ return ret;
+ }
+ for (int i = 0; i < write_index_; i++) {
+ write_bits(delta_arr_[i], bit_width, inner);
+ }
+ flush_remaining(inner);
+ reset();
+
+ const bool overflow = has_overflow();
+ if (overflow) {
+ std::vector<uint8_t> underflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ std::vector<uint8_t> value_overflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ bool has_value_overflow = false;
+ for (int i = 0; i < num_values; i++) {
+ int8_t f = underflow_flags_[static_cast<size_t>(i)];
+ if (f == 1) {
+ underflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ } else if (f == -1) {
+ has_value_overflow = true;
+ value_overflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ }
+ }
+ constexpr uint32_t kJavaOverflowMagic =
+ 2147483647u; // Integer.MAX_VALUE
+ constexpr uint32_t kJavaValueOverflowMagic =
+ 2147483646u; // Integer.MAX_VALUE - 1
Review Comment:
Renamed as suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix TS_2DIFF float/double page compatibility [tsfile]
Posted by "hongzhi-gao (via GitHub)" <gi...@apache.org>.
hongzhi-gao commented on code in PR #796:
URL: https://github.com/apache/tsfile/pull/796#discussion_r3354527787
##########
cpp/src/encoding/ts2diff_encoder.h:
##########
@@ -402,5 +484,168 @@ FORCE_INLINE int DoubleTS2DIFFEncoder::encode(double value,
return do_encode(value, out);
}
+// Keep float/double TS_2DIFF page layout compatible with Java.
+FORCE_INLINE int FloatTS2DIFFEncoder::flush(common::ByteStream& out_stream) {
+ int ret = common::E_OK;
+ if (write_index_ == -1) {
+ return common::E_OK;
+ }
+ const int num_values = write_index_ + 1;
+ common::ByteStream inner(1024, common::MOD_TS2DIFF_OBJ, false);
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(max_point_number_), inner))) {
+ return ret;
+ }
+ SIMDOps<int32_t>::rebase(delta_arr_, delta_arr_min_, write_index_);
+ int bit_width = cal_bit_width(delta_arr_max_ - delta_arr_min_);
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(write_index_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(bit_width), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(delta_arr_min_), inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_ui32(
+ static_cast<uint32_t>(first_value_), inner))) {
+ return ret;
+ }
+ for (int i = 0; i < write_index_; i++) {
+ write_bits(delta_arr_[i], bit_width, inner);
+ }
+ flush_remaining(inner);
+ reset();
+
+ const bool overflow = has_overflow();
+ if (overflow) {
+ std::vector<uint8_t> underflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ std::vector<uint8_t> value_overflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ bool has_value_overflow = false;
+ for (int i = 0; i < num_values; i++) {
+ int8_t f = underflow_flags_[static_cast<size_t>(i)];
+ if (f == 1) {
+ underflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ } else if (f == -1) {
+ has_value_overflow = true;
+ value_overflow_bitmap[static_cast<size_t>(i / 8)] |=
+ static_cast<uint8_t>(1u << (i % 8));
+ }
+ }
+ constexpr uint32_t kJavaOverflowMagic =
+ 2147483647u; // Integer.MAX_VALUE
+ constexpr uint32_t kJavaValueOverflowMagic =
+ 2147483646u; // Integer.MAX_VALUE - 1
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ has_value_overflow ? kJavaValueOverflowMagic
+ : kJavaOverflowMagic,
+ out_stream))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(num_values), out_stream))) {
+ return ret;
+ }
+ const uint32_t bm_len = static_cast<uint32_t>(num_values / 8 + 1);
+ if (RET_FAIL(out_stream.write_buf(underflow_bitmap.data(), bm_len))) {
+ return ret;
+ }
+ if (has_value_overflow && RET_FAIL(out_stream.write_buf(
+ value_overflow_bitmap.data(), bm_len))) {
+ return ret;
+ }
+ }
+ if (RET_FAIL(merge_byte_stream(out_stream, inner, true))) {
+ return ret;
+ }
+ underflow_flags_.clear();
+ return ret;
+}
+
+FORCE_INLINE int DoubleTS2DIFFEncoder::flush(common::ByteStream& out_stream) {
+ int ret = common::E_OK;
+ if (write_index_ == -1) {
+ return common::E_OK;
+ }
+ const int num_values = write_index_ + 1;
+ common::ByteStream inner(1024, common::MOD_TS2DIFF_OBJ, false);
+ if (RET_FAIL(common::SerializationUtil::write_var_uint(
+ static_cast<uint32_t>(max_point_number_), inner))) {
+ return ret;
+ }
+ SIMDOps<int64_t>::rebase(delta_arr_, delta_arr_min_, write_index_);
+ int bit_width = cal_bit_width(delta_arr_max_ - delta_arr_min_);
+ if (RET_FAIL(common::SerializationUtil::write_i32(write_index_, inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_i32(bit_width, inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_i64(delta_arr_min_, inner))) {
+ return ret;
+ }
+ if (RET_FAIL(common::SerializationUtil::write_i64(first_value_, inner))) {
+ return ret;
+ }
+ for (int i = 0; i < write_index_; i++) {
+ write_bits(delta_arr_[i], bit_width, inner);
+ }
+ flush_remaining(inner);
+ reset();
+
+ const bool overflow = has_overflow();
+ if (overflow) {
+ std::vector<uint8_t> underflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
+ std::vector<uint8_t> value_overflow_bitmap(
+ static_cast<size_t>(num_values / 8 + 1), 0);
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix TS_2DIFF float/double page compatibility [tsfile]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #796:
URL: https://github.com/apache/tsfile/pull/796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org