You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tsfile.apache.org by co...@apache.org on 2026/10/10 10:37:25 UTC
(tsfile) branch rc/2.5.0 updated (e6d09dd73 -> 9ec6d2380)
This is an automated email from the ASF dual-hosted git repository.
ColinLeeo pushed a change to branch rc/2.5.0
in repository https://gitbox.apache.org/repos/asf/tsfile.git
from e6d09dd73 fix(query): restore empty results for inverted time ranges (#954)
add 5e207cd02 Cache type service readers in page readers (#955)
add 901c24ad0 Restore custom encryption implementation compatibility (#956)
add 98335815e Add CodeQL analysis for Go code (#957)
add 8fdbc49bc Complete TimeColumn accessors and fix dictionary primitive lookup (#961)
add 9391fc160 fix(cpp): align the ANTLR4 compatibility bound with the enforced range (#965)
add 9b40e40c0 perf(python): accelerate Dataset Index metadata lookups (#959)
add e3ecbddca Move type dispatch outside batch read and write loops (#963)
add 83fe5594f fix(cpp): flush pending data on writer close (#969)
add 09899a0c1 Add sketch command in tsfile-cli from `TsFileSketchTool` (#838)
add 3cf7a0f63 fix: update sketch layout (#978)
add 05b5d939c fix(cpp): stop misreading pages after an empty value page (#976)
add 4cabc26ea build(deps): bump actions/setup-java from 5.6.0 to 6.0.1 (#981)
add 4177a89dd build(deps): bump github/codeql-action from 4.37.9 to 4.38.2 (#982)
add 4e41a7b00 build(deps): bump actions/setup-go from 6 to 7 (#983)
add 7b1c127b7 build(deps): bump com.fasterxml.jackson.core:jackson-core in /java (#985)
add 9ed0f9400 build(deps): bump com.fasterxml.jackson.core:jackson-databind in /java (#984)
add e704c475b Bump com.gradle:develocity-maven-extension from 1.23.1 to 2.4.1 (#828)
add f4691f041 Update develocity-maven-extension version to 2.6.0 (#986)
add 7042e3f8b Change vulnerability-checks fail configuration to false in pom.xml (#987)
add c00145b22 Remove unavailable OSS Index vulnerability check (#988)
add a2b9369ff fix(cpp): reject special input files before open to avoid FIFO hang (#977)
add b9a9cfa32 fix(cpp): preserve table time statistics in filtered metadata (#989)
add 9cd767df5 fix: add scanning patch (#975)
add fc5e1cfc3 Remove the unused maven-dependency-tree enforcer dependency (#990)
add fce6c421e fix(cpp): escape control characters in tsfile-cli table output (#974)
add 47c0ec4e9 fix(cpp): restore reader test file cleanup (#992)
add 617b24004 docs: align root tsfile-cli skill with current CLI (#995)
add 133c9b63b fix(cpp): replace malformed UTF-8 in CLI text output (#997)
add 9f35d7613 fix(cpp): preserve leading backslashes in tsfile-cli CSV round trips (#994)
add 3f7228590 build(deps): bump at.yawk.lz4:lz4-java from 1.11.1 to 1.11.4 in /java (#991)
add 0e2ceed74 fix(cpp): emit bare measurement names for tree-model result columns (#993)
add d33e640d3 fix(python): guard dataset index reads after close (#999)
add 1d87fe0b0 refactor: remove IDeviceID.isTableModel (#998)
add 2cf14711a fix(cpp): propagate table read failures without silent data loss (#970)
add 51a114b1a fix(cpp): keep aligned value columns row-aligned when a record omits measurements (#968)
add bd2b86ace fix(cpp): preserve quoted CSV line endings in tsfile-cli (#996)
add e0f736699 fix(cpp): correct aligned all-null pages and page advancement (#1001)
add 01c3fe329 fix(cpp): reject UTF-8 name collisions before CLI output (#1002)
add dfcb04760 [Python] Trust dataset indexes and expose bounded read caches (#1004)
add 51b04eb74 fix(cpp): apply residual row offsets in non-aligned page decoding (#1007)
new 9ec6d2380 docs: add release notes for TsFile 2.5.0
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/code-coverage.yml | 2 +-
.github/workflows/codeql.yml | 43 +-
.github/workflows/compatibility-test.yml | 4 +-
.github/workflows/cppcheck.yml | 2 +-
.github/workflows/unit-test-cpp-msvc.yml | 2 +-
.github/workflows/unit-test-cpp.yml | 2 +-
.github/workflows/unit-test-go.yml | 4 +-
.github/workflows/unit-test-java.yml | 2 +-
.github/workflows/unit-test-python-msvc.yml | 2 +-
.github/workflows/unit-test-python.yml | 2 +-
.github/workflows/wheels.yml | 6 +-
.mvn/extensions.xml | 2 +-
RELEASE_NOTES.md | 30 +
cpp/README.md | 2 +-
cpp/cmake/tests/ANTLR4DependencyTest.cmake | 2 +-
cpp/examples/cpp_examples/demo_read.cpp | 4 +-
cpp/src/common/tsfile_common.h | 6 +
cpp/src/cwrapper/tsfile_cwrapper.cc | 1101 ++++++++++++--------
cpp/src/cwrapper/tsfile_cwrapper.h | 120 ++-
cpp/src/file/local_random_access_read_file.cc | 17 +
cpp/src/file/restorable_tsfile_io_writer.cc | 8 +
cpp/src/file/tsfile_io_reader.cc | 23 +-
cpp/src/reader/aligned_chunk_reader.cc | 45 +-
.../reader/block/device_ordered_tsblock_reader.cc | 5 +-
cpp/src/reader/chunk_reader.cc | 165 +--
cpp/src/reader/chunk_reader.h | 19 +-
cpp/src/reader/device_meta_iterator.cc | 60 +-
cpp/src/reader/device_meta_iterator.h | 11 +-
cpp/src/reader/imeta_data_querier.h | 2 +-
cpp/src/reader/meta_data_querier.cc | 6 +-
cpp/src/reader/meta_data_querier.h | 5 +-
cpp/src/reader/prepared_series.h | 5 +-
cpp/src/reader/table_query_executor.cc | 25 +-
cpp/src/reader/table_result_set.cc | 20 +
cpp/src/reader/table_result_set.h | 4 +
cpp/src/reader/task/device_task_iterator.cc | 4 +-
cpp/src/reader/task/device_task_iterator.h | 2 +-
cpp/src/reader/tsfile_reader.cc | 144 ++-
cpp/src/reader/tsfile_reader.h | 50 +
cpp/src/reader/tsfile_series_scan_iterator.cc | 13 +
cpp/src/writer/tsfile_table_writer.h | 2 +-
cpp/src/writer/tsfile_tree_writer.h | 8 +-
cpp/src/writer/tsfile_writer.cc | 261 +++--
cpp/src/writer/tsfile_writer.h | 22 +-
cpp/src/writer/value_chunk_writer.cc | 4 +
cpp/src/writer/value_chunk_writer.h | 35 +
cpp/src/writer/value_page_writer.cc | 24 +-
cpp/src/writer/value_page_writer.h | 20 +
cpp/test/common/tsfile_common_test.cc | 41 +
cpp/test/cwrapper/c_release_test.cc | 35 +-
cpp/test/cwrapper/cwrapper_test.cc | 42 +
cpp/test/file/restorable_tsfile_io_writer_test.cc | 204 ++++
cpp/test/reader/aligned_null_page_test.cc | 235 +++++
cpp/test/reader/prepared_series_test.cc | 180 +++-
cpp/test/reader/table_read_failure_test.cc | 1005 ++++++++++++++++++
...odel_encoding_compression_compatibility_test.cc | 149 ++-
.../reader/table_view/tsfile_reader_table_test.cc | 151 ++-
cpp/test/reader/tsfile_reader_test.cc | 23 +-
cpp/test/tools/cli_requirements_test.cc | 294 +++++-
cpp/test/tools/cli_test_util.h | 38 +
cpp/test/tools/command_e2e_test.cc | 497 ++++++++-
cpp/test/tools/fixtures/complex_table.tsfile | Bin 0 -> 5364 bytes
cpp/test/tools/fixtures/complex_tree.tsfile | Bin 0 -> 1785 bytes
cpp/test/tools/golden/category_edge_sketch.txt | 18 +-
cpp/test/tools/golden/empty_field_meta_csv.txt | 2 +-
cpp/test/tools/golden/empty_field_meta_ndjson.txt | 2 +-
cpp/test/tools/golden/empty_field_meta_table.txt | 2 +-
cpp/test/tools/golden/empty_field_sketch.txt | 53 +-
cpp/test/tools/golden/empty_table_sketch.txt | 18 +-
cpp/test/tools/golden/empty_tree_sketch.txt | 17 +-
cpp/test/tools/golden/table_sketch.txt | 581 ++++++++++-
cpp/test/tools/golden/table_stats_csv.txt | 2 +-
cpp/test/tools/golden/table_stats_ndjson.txt | 2 +-
cpp/test/tools/golden/table_stats_table.txt | 2 +-
cpp/test/tools/golden/tree_cat_csv.txt | 2 +-
cpp/test/tools/golden/tree_cat_ndjson.txt | 4 +-
cpp/test/tools/golden/tree_cat_table.txt | 6 +-
cpp/test/tools/golden/tree_export_csv.txt | 2 +-
cpp/test/tools/golden/tree_export_ndjson.txt | 40 +-
cpp/test/tools/golden/tree_export_table.txt | 42 +-
cpp/test/tools/golden/tree_head_csv.txt | 2 +-
cpp/test/tools/golden/tree_head_ndjson.txt | 4 +-
cpp/test/tools/golden/tree_head_table.txt | 6 +-
cpp/test/tools/golden/tree_sketch.txt | 148 ++-
cpp/test/tools/input_format_test.cc | 38 +-
cpp/test/tools/model_format_e2e_test.cc | 66 +-
cpp/test/tools/output_format_test.cc | 185 ++++
cpp/test/tools/utf8_collision_e2e_test.cc | 438 ++++++++
.../writer/table_view/tsfile_writer_table_test.cc | 22 +-
cpp/test/writer/tsfile_writer_test.cc | 691 +++++++++++-
cpp/test/writer/value_chunk_writer_test.cc | 63 +-
cpp/test/writer/value_page_writer_test.cc | 33 +
cpp/third_party/README.md | 2 +-
cpp/tools/README.md | 40 +-
cpp/tools/cli/run_cli.cc | 48 +-
cpp/tools/commands/cmd_count.cc | 40 +-
cpp/tools/commands/cmd_export.cc | 65 +-
cpp/tools/commands/cmd_ls.cc | 51 +-
cpp/tools/commands/cmd_meta.cc | 6 +-
cpp/tools/commands/cmd_schema.cc | 16 +-
cpp/tools/commands/cmd_sketch.cc | 600 +++++++++++
cpp/tools/commands/cmd_stats.cc | 89 +-
cpp/tools/commands/cmd_write.cc | 22 +-
cpp/tools/commands/commands.h | 22 +-
cpp/tools/commands/row_query.cc | 223 ++--
cpp/tools/commands/sketch_layout.cc | 946 +++++++++++++++++
cpp/tools/commands/sketch_layout.h | 221 ++++
cpp/tools/commands/statistics.cc | 43 +-
cpp/tools/commands/statistics.h | 4 +-
cpp/tools/format/input_format.cc | 24 +-
cpp/tools/format/input_format.h | 12 +-
cpp/tools/format/output_format.cc | 171 ++-
cpp/tools/format/output_format.h | 29 +
cpp/tools/format/result_set_format.cc | 21 +-
cpp/tools/format/result_set_format.h | 5 +-
cpp/tools/tools_main.cc | 21 +-
java/pom.xml | 6 +-
.../org/apache/tsfile/tools/ArrowSourceReader.java | 123 ++-
.../apache/tsfile/tools/ParquetSourceReader.java | 91 +-
.../org/apache/tsfile/tools/TabletBuilder.java | 60 +-
.../org/apache/tsfile/tools/ValueConverter.java | 17 +
.../org/apache/tsfile/tools/TabletBuilderTest.java | 105 ++
.../codegen/templates/PageDataReaderTemplate.ftl | 205 ++++
.../templates/TabletColumnWriterTemplate.ftl | 72 ++
.../org/apache/tsfile/encrypt/EncryptUtils.java | 20 +-
.../org/apache/tsfile/file/metadata/IDeviceID.java | 2 -
.../apache/tsfile/file/metadata/PlainDeviceID.java | 5 -
.../tsfile/file/metadata/StringArrayDeviceID.java | 6 -
.../read/common/block/column/DictionaryColumn.java | 2 +-
.../read/common/block/column/TimeColumn.java | 43 +-
.../reader/block/SingleDeviceTsBlockReader.java | 14 +-
.../apache/tsfile/read/reader/page/PageReader.java | 67 +-
.../tsfile/read/reader/page/ValuePageReader.java | 103 +-
.../java/org/apache/tsfile/utils/TypeServices.java | 130 +++
.../write/chunk/AlignedChunkGroupWriterImpl.java | 27 +-
.../chunk/NonAlignedChunkGroupWriterImpl.java | 41 +-
.../tsfile/write/chunk/TabletWriteContext.java | 68 ++
.../tsfile/write/chunk/ValueChunkWriter.java | 5 +
.../apache/tsfile/write/writer/TsFileIOWriter.java | 4 +-
...eModelEncodingCompressionCompatibilityTest.java | 218 +++-
.../tsfile/custom/encrypt/ExternalEncryptTest.java | 126 +++
.../org/apache/tsfile/encrypt/EncryptTest.java | 4 +-
.../apache/tsfile/file/metadata/IDeviceIDTest.java | 10 -
.../org/apache/tsfile/read/common/ColumnTest.java | 61 ++
.../read/reader/PageDataBatchReaderTest.java | 272 +++++
.../tsfile/read/reader/PageDataTestSupport.java | 139 +++
.../tsfile/write/chunk/AlignedNullWriteTest.java | 155 +++
.../tsfile/write/chunk/TabletColumnWriteTest.java | 164 +++
pom.xml | 82 +-
python/README-zh.md | 61 ++
python/README.md | 67 ++
python/setup.py | 5 +
python/tests/test_dataset_index.py | 521 ++++++++-
python/tests/test_dataset_read_options.py | 524 ++++++++++
python/tests/test_reader_metadata.py | 44 +
python/tests/test_reader_sources.py | 115 +-
python/tests/test_tag_filter_query.py | 23 +
python/tests/test_tsfile_dataset.py | 32 +
python/tsfile/dataset/_config.py | 66 ++
python/tsfile/dataset/_index.pyx | 537 ++++++++++
python/tsfile/dataset/dataframe.py | 45 +-
python/tsfile/dataset/index.py | 130 ++-
python/tsfile/dataset/runtime.py | 462 +++++---
python/tsfile/tsfile_cpp.pxd | 10 +-
python/tsfile/tsfile_py_cpp.pxd | 4 +-
python/tsfile/tsfile_py_cpp.pyx | 50 +-
python/tsfile/tsfile_reader.pyx | 63 +-
skills/tsfile-cli/SKILL.md | 140 ++-
168 files changed, 13997 insertions(+), 1802 deletions(-)
create mode 100644 cpp/test/reader/aligned_null_page_test.cc
create mode 100644 cpp/test/reader/table_read_failure_test.cc
create mode 100644 cpp/test/tools/fixtures/complex_table.tsfile
create mode 100644 cpp/test/tools/fixtures/complex_tree.tsfile
create mode 100644 cpp/test/tools/utf8_collision_e2e_test.cc
create mode 100644 cpp/tools/commands/cmd_sketch.cc
create mode 100644 cpp/tools/commands/sketch_layout.cc
create mode 100644 cpp/tools/commands/sketch_layout.h
create mode 100644 java/tsfile/src/main/codegen/templates/PageDataReaderTemplate.ftl
create mode 100644 java/tsfile/src/main/codegen/templates/TabletColumnWriterTemplate.ftl
create mode 100644 java/tsfile/src/main/java/org/apache/tsfile/write/chunk/TabletWriteContext.java
create mode 100644 java/tsfile/src/test/java/org/apache/tsfile/custom/encrypt/ExternalEncryptTest.java
create mode 100644 java/tsfile/src/test/java/org/apache/tsfile/read/reader/PageDataBatchReaderTest.java
create mode 100644 java/tsfile/src/test/java/org/apache/tsfile/read/reader/PageDataTestSupport.java
create mode 100644 java/tsfile/src/test/java/org/apache/tsfile/write/chunk/AlignedNullWriteTest.java
create mode 100644 java/tsfile/src/test/java/org/apache/tsfile/write/chunk/TabletColumnWriteTest.java
create mode 100644 python/tests/test_dataset_read_options.py
create mode 100644 python/tsfile/dataset/_config.py
create mode 100644 python/tsfile/dataset/_index.pyx
(tsfile) 01/01: docs: add release notes for TsFile 2.5.0
Posted by co...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ColinLeeo pushed a commit to branch rc/2.5.0
in repository https://gitbox.apache.org/repos/asf/tsfile.git
commit 9ec6d238083f4440bb58511f437b3211f00c2b47
Author: ColinLee <sh...@163.com>
AuthorDate: Sat Oct 10 18:37:04 2026 +0800
docs: add release notes for TsFile 2.5.0
---
RELEASE_NOTES.md | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 81ca4c67a..c136168bb 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -19,6 +19,36 @@
-->
+# Apache TsFile 2.5.0
+
+## New Features
+
+- [Go] Added table-model read and write APIs backed by the C++ library, with Tablet and Arrow batch writes, time-range and TAG filtering, pagination, and Arrow batch reads. (#929, #949)
+- [Python] Added opt-in persistent dataset indexes and a native query runtime for `TsFileDataFrame`, supporting indexed row slicing and time-range queries across multiple TsFile files. (#911)
+- [Java/C++/Python] Added read and write support for binary file-level properties. (#882, #897)
+- [Java] Added optional table point counting and a tool to inspect and repair stored table point counts. (#882)
+- [C++/Python] Added seekable random-access data sources, including Python binary file objects and remote files opened through libraries such as `fsspec`. (#939)
+- [C++/Python] Added an optional memory-mapped local file read backend with `PREAD`, `MMAP`, and `AUTO` modes. (#920)
+- [C++] Added CHIMP, RLBE, and CAMEL encodings, Zstandard compression, and optional LZMA2 compression, with Java/C++ compatibility coverage. (#905)
+- [C++] Expanded `tsfile-cli` with tree-model and table-model inspection, filtered reads, CSV/NDJSON export, CSV import, and a physical file `sketch` command. (#915, #838)
+- [Tools] Added reusable agent skills for TsFile development and `tsfile-cli` workflows. (#907)
+
+## Improvements
+
+- [Python] Accelerated dataset-index metadata lookups with Cython and descriptor caching, added configurable read-cache and concurrency limits, and added an index-trust option for immutable datasets. (#959, #1004)
+- [C++] Improved batch read throughput with faster Gorilla decoding, bulk column output, and aligned row-offset pushdown. (#873, #898, #863)
+- [C++] Added static-library builds and configurable system or verified source-archive dependency sourcing, including offline builds from cached archives. (#894, #905)
+- [Java] Added buffered input and configurable `TsFileInput` construction for readers. (#868, #869)
+- [Java] Streamlined data-type handling, cached page-reader type services, and moved type dispatch outside batch read and write loops. (#872, #955, #963)
+- [Java] Reduced query merge overhead with primitive long heaps and sets, and optimized small bitmaps with a long-backed implementation and range-query APIs. (#906, #871, #884)
+
+## Bug Fixes
+
+- [C++/Python] Fixed silent data loss from table-read failures by propagating metadata, decoding, and I/O errors to callers. (#970)
+- [C++] Fixed aligned null-page handling, row alignment when records omit measurements, and flushing of pending data when closing writers. (#968, #969, #976, #1001)
+- [C++] Fixed TS_2DIFF floating-point compatibility, wide INT64 decoding, and Gorilla sentinel-value handling. (#901, #947, #883)
+- [Java] Restored compatibility with custom encryption implementations. (#956)
+
# Apache TsFile 2.4.0
## New Features