You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@tsfile.apache.org by "SamM755 (via GitHub)" <gi...@apache.org> on 2026/10/09 07:24:33 UTC
[PR] [research/cluster-compress] Add joint Cluster compression for aligned numeric series [tsfile]
SamM755 opened a new pull request, #1000:
URL: https://github.com/apache/tsfile/pull/1000
## Description
Target: `apache/tsfile:research/cluster-compress`.
Integrate the column-selection + multidimensional ACluster/KCluster + ALP pipeline into native aligned numeric pages. An aligned group marked ACLUSTER or KCLUSTER can be written and queried as typed numeric columns, with timestamp-value associations and cross-column correspondence preserved through a shared row reordering.
Within each writer, columns carrying the same Cluster encoding form a candidate group. The CV-elbow selector fits on the first block; selected, safely scaled columns use joint clustering, while the remaining numeric value columns use the Java ALP path with exact exceptions and RAW fallback. This ALP implementation uses the project's table-block format rather than upstream ALP wire bytes. Timestamp keys remain separately stored as exact INT64 keys and are not ALP encoded; the aligned shared time column retains native TsFile encoding.
The change also repairs the research branch's scalar Cluster stream, adds timestamp-aware scalar SQL pages, and retains the separate complete-record BLOB API. Native page readers expose values in timestamp-key order for the existing SQL scan interface. No original row-position or inverse-permutation array is persisted.
### Companion IoTDB integration
The companion contribution targets `apache/iotdb:research/cluster-compress`, consumes this branch's Java `common`/`tsfile` artifacts at `2.1.0-SNAPSHOT`, and adds live aligned/scalar SQL checks. All nodes reading these research pages need the matching modified TsFile reader. See `java/tsfile/ALIGNED_CLUSTER.md` for SQL examples, format details, limits, and build commands.
### Validation
- 72 distinct targeted TsFile unit/regression cases passed: ClusterCodecTest (32), ClusterTableCodecTest (20), ClusterAlignedTsFileTest (8), PageWriterTest (6), and ValuePageWriterTest (6). Checkstyle, Spotless, and license checks passed.
- The companion IoTDB Session module passed 3 unit tests; DataNode compiled against the modified dependency.
- On an isolated IoTDB 2.0.2-SNAPSHOT server at the research baseline, each method passed 1,200 aligned rows and 400 scalar rows using ordinary SQL. Checks cover exact timestamp/value pairs, projection, time/value predicates, descending scans, and aggregates.
- A forced DataNode restart recovered 200 unflushed aligned rows per method from WAL, and the read-only SQL checks passed again. The retained BLOB API also verified 1,200 complete records per method before and after restart.
Validation was performed on 2026-10-08. Full-repository Failsafe tests were not run because the offline Maven cache lacked the provider; the targeted build used `-Dtsfile.it.skip=true`, with the live SQL checks executed separately.
### Research scope
This PR is for the research branch. Column projections currently repeat timestamp keys and required metadata to support independent decoding; native file sizes are therefore distinct from the benchmark container's compression ratio. Compaction variants, deletion/overwrite, schema evolution, replication, and full-scale performance are not claimed as comprehensively validated here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [research/cluster-compress] Add joint Cluster compression for aligned numeric series [tsfile]
Posted by "ColinLeeo (via GitHub)" <gi...@apache.org>.
ColinLeeo merged PR #1000:
URL: https://github.com/apache/tsfile/pull/1000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org