You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tsfile.apache.org by "jixuan1989 (via GitHub)" <gi...@apache.org> on 2026/10/17 10:09:43 UTC
[I] [Bug][C++] TsFile writer ignores max_point_number for floating-point TS_2DIFF [tsfile]
jixuan1989 opened a new issue, #1020:
URL: https://github.com/apache/tsfile/issues/1020
## English
### Problem and shared input
With the same TS_2DIFF measurement properties and data, the Java writer applies `max_point_number`, while the C++ writer continues to use two decimal places. Both writers finish successfully and their files are readable, but the stored values differ. Requesting precision 4 therefore loses precision that Java preserves.
```json
{
"model": "TREE",
"device": "root.precision.d",
"measurement": "v",
"type": "FLOAT",
"encoding": "TS_2DIFF",
"compression": "UNCOMPRESSED",
"properties": {"max_point_number": "0"},
"timestamp": 1,
"value": 1.23456
}
```
### Actual input and output
Java writes and reads its own file:
```text
language=Java mode=write type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.000000
read=SUCCESS rows=1
```
Current C++ writes and reads its own file:
```text
language=C++ mode=write type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.230000
read=SUCCESS rows=1
```
Both processes exit `0`. The difference concerns the stored value, not a read failure. For C++, every checked operation returns `E_OK`; Java's writer return value is a flush-threshold indicator rather than an `E_OK` status.
Java reads the C++ file:
```text
language=Java mode=read type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.230000
read=SUCCESS rows=1
```
C++ reads the Java file:
```text
language=C++ mode=read type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.000000
read=SUCCESS rows=1
```
The cross-reads preserve each writer's output. The writer-side precision setting causes the difference; both decoders handle these files.
### Additional cases and controls
| Type / alignment | max_point_number | Java writes → either reader | C++ writes → either reader |
| --- | --- | --- | --- |
| FLOAT and DOUBLE; nonaligned and aligned | 0 | 1.000000 | 1.230000 |
| FLOAT and DOUBLE; nonaligned and aligned | 4 | 1.234600 | 1.230000 |
| FLOAT and DOUBLE; nonaligned and aligned | 2 | 1.230000 | 1.230000 |
| FLOAT and DOUBLE; nonaligned and aligned | default (property absent) | 1.230000 | 1.230000 |
The script checks 16 writer configurations: two types × two alignments × four precision settings. It confirms 8 mismatches, 8 matching controls and 32 successful cross-reads. Values above are printed with six decimal places; this is decoded-value comparison, not whole-file byte equality.
### Reproduction environment and command
Both implementations were freshly built from upstream commit `12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f` (`develop`, `2.5.1-SNAPSHOT`). Tested on Windows x64, MSVC 14.50, Release static C++ library and JDK 24.0.2. Prerequisites: JDK 17+, Maven, CMake and Visual Studio C++ build tools. If Java is not on PATH, set `JAVA_HOME` to the JDK directory.
Save all four complete files below in a directory and run PowerShell from that directory against a clean checkout of this commit:
```powershell
.\reproduce.ps1 -TsFileSource D:\code\tsfile -BuildRoot .\build-precision
```
Use a fresh `BuildRoot` for each run. The script exits `0` only if every case matches the reported behavior; that means the defect was reproduced. The final line must be `reproduction=CONFIRMED`. Optional codecs are disabled and both time/value columns are uncompressed. C++ disables automatic table-schema generation for this tree-model reproducer, isolating this defect from #1017. The C++ program uses the vector-of-schema-pointers registration overload so the caller's properties remain available in the registered schema.
### Complete reproduction sources
Save as `PrecisionRepro.java`:
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
* either express or implied. See the License for the specific
* language governing permissions and limitations under the License.
*/
import org.apache.tsfile.common.conf.TSFileDescriptor;
import org.apache.tsfile.enums.TSDataType;
import org.apache.tsfile.file.metadata.enums.CompressionType;
import org.apache.tsfile.file.metadata.enums.TSEncoding;
import org.apache.tsfile.read.TsFileReader;
import org.apache.tsfile.read.TsFileSequenceReader;
import org.apache.tsfile.read.common.Path;
import org.apache.tsfile.read.common.RowRecord;
import org.apache.tsfile.read.expression.QueryExpression;
import org.apache.tsfile.read.query.dataset.QueryDataSet;
import org.apache.tsfile.write.TsFileWriter;
import org.apache.tsfile.write.record.TSRecord;
import org.apache.tsfile.write.schema.MeasurementSchema;
import java.io.File;
import java.util.List;
import java.util.Locale;
import java.util.Map;
public class PrecisionRepro {
public static void main(String[] args) throws Exception {
if (args.length != 5) throw new IllegalArgumentException("write|read file FLOAT|DOUBLE default|precision aligned|nonaligned");
TSDataType type = TSDataType.valueOf(args[2]);
boolean aligned = args[4].equals("aligned");
File file = new File(args[1]);
TSFileDescriptor.getInstance().getConfig().setCompressor("UNCOMPRESSED");
TSFileDescriptor.getInstance().getConfig().setFloatPrecision(2);
if (args[0].equals("write")) {
Map<String, String> props = args[3].equals("default") ? Map.of() : Map.of("max_point_number", args[3]);
MeasurementSchema schema = new MeasurementSchema("v", type, TSEncoding.TS_2DIFF, CompressionType.UNCOMPRESSED, props);
try (TsFileWriter writer = new TsFileWriter(file)) {
if (aligned) writer.registerAlignedTimeseries(new Path("root.precision.d"), List.of(schema));
else writer.registerTimeseries(new Path("root.precision.d"), schema);
TSRecord record = new TSRecord("root.precision.d", 1L);
if (type == TSDataType.FLOAT) record.addPoint("v", 1.23456f);
else record.addPoint("v", 1.23456d);
writer.writeRecord(record);
}
}
System.out.printf("language=Java mode=%s type=%s precision=%s alignment=%s%n", args[0], type, args[3], args[4]);
System.out.println("input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED");
try (TsFileReader reader = new TsFileReader(new TsFileSequenceReader(file.getPath()))) {
QueryDataSet result = reader.query(QueryExpression.create(List.of(new Path("root.precision.d", "v", true)), null));
int rows = 0;
while (result.hasNext()) {
RowRecord row = result.next();
double value = type == TSDataType.FLOAT ? row.getFields().get(0).getFloatV() : row.getFields().get(0).getDoubleV();
System.out.printf(Locale.ROOT, "output.timestamp=%d output.value=%.6f%n", row.getTimestamp(), value);
if (row.getTimestamp() != 1) throw new AssertionError("timestamp");
rows++;
}
if (rows != 1) throw new AssertionError("row count=" + rows);
System.out.println("read=SUCCESS rows=" + rows);
}
}
}
```
Save as `precision_repro.cc`:
```cpp
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
* either express or implied. See the License for the specific
* language governing permissions and limitations under the License.
*/
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <string>
#include <vector>
#include "reader/tsfile_reader.h"
#include "writer/tsfile_writer.h"
void check(int code, const char* operation) {
if (code != common::E_OK)
throw std::runtime_error(std::string(operation) + " code=" + std::to_string(code));
}
int run(int argc, char** argv) {
if (argc != 6) throw std::runtime_error("write|read file FLOAT|DOUBLE default|precision aligned|nonaligned");
const bool is_float = std::string(argv[3]) == "FLOAT";
const bool aligned = std::string(argv[5]) == "aligned";
const auto type = is_float ? common::FLOAT : common::DOUBLE;
common::g_config_value_.time_compress_type_ = common::UNCOMPRESSED;
if (std::string(argv[1]) == "write") {
storage::TsFileWriter writer;
check(writer.open(argv[2]), "open writer");
writer.set_generate_table_schema(false);
auto* schema = new storage::MeasurementSchema("v", type, common::TS_2DIFF, common::UNCOMPRESSED);
if (std::string(argv[4]) != "default") schema->props_["max_point_number"] = argv[4];
std::vector<storage::MeasurementSchema*> schemas{schema};
if (aligned) check(writer.register_aligned_timeseries("root.precision.d", schemas), "register aligned");
else check(writer.register_timeseries("root.precision.d", schemas), "register nonaligned");
storage::TsRecord record(1, "root.precision.d");
if (is_float) check(record.add_point("v", 1.23456f), "add float");
else check(record.add_point("v", 1.23456), "add double");
check(aligned ? writer.write_record_aligned(record) : writer.write_record(record), "write record");
check(writer.close(), "close writer");
}
std::cout << "language=C++ mode=" << argv[1] << " type=" << argv[3]
<< " precision=" << argv[4] << " alignment=" << argv[5] << '\n'
<< "input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED\n";
storage::TsFileReader reader;
check(reader.open(argv[2]), "open reader");
storage::ResultSet* result = nullptr;
std::vector<std::string> paths{"root.precision.d.v"};
check(reader.query(paths, 0, 2, result), "query");
int rows = 0;
for (;;) {
bool has_next = false;
check(result->next(has_next), "next");
if (!has_next) break;
double value = is_float ? result->get_value<float>(2) : result->get_value<double>(2);
int64_t timestamp = result->get_value<int64_t>(1);
std::cout << "output.timestamp=" << timestamp << " output.value=" << std::fixed << std::setprecision(6) << value << '\n';
if (timestamp != 1) throw std::runtime_error("timestamp");
++rows;
}
reader.destroy_query_data_set(result);
check(reader.close(), "close reader");
if (rows != 1) throw std::runtime_error("row count=" + std::to_string(rows));
std::cout << "read=SUCCESS rows=" << rows << '\n';
return 0;
}
int main(int argc, char** argv) {
try {
check(storage::libtsfile_init(), "initialize");
int code = run(argc, argv);
storage::libtsfile_destroy();
return code;
} catch (const std::exception& error) {
std::cerr << error.what() << '\n';
return 2;
}
}
```
### Complete build and run files
Save `CMakeLists.txt` and `reproduce.ps1` beside the two programs. The optional `StaticLibrary` and `JavaClasspath` arguments permit existing builds; omitting them builds both implementations from the supplied checkout.
`CMakeLists.txt`:
```cmake
cmake_minimum_required(VERSION 3.15)
project(tsfile_precision_repro LANGUAGES CXX)
set(TSFILE_SOURCE "" CACHE PATH "Apache TsFile source checkout")
set(TSFILE_LIBRARY "" CACHE FILEPATH "Release static TsFile library")
set(TSFILE_BASELINE_INCLUDE "" CACHE PATH "Optional preserved baseline headers")
add_executable(precision_repro precision_repro.cc)
target_compile_features(precision_repro PRIVATE cxx_std_14)
target_compile_definitions(precision_repro PRIVATE TSFILE_STATIC NOMINMAX ENABLE_MEM_STAT ENABLE_THREADS)
target_include_directories(precision_repro PRIVATE "${TSFILE_SOURCE}/cpp/src")
if(TSFILE_BASELINE_INCLUDE)
target_include_directories(precision_repro BEFORE PRIVATE "${TSFILE_BASELINE_INCLUDE}")
endif()
target_link_libraries(precision_repro PRIVATE "${TSFILE_LIBRARY}")
if(MSVC)
target_compile_options(precision_repro PRIVATE /utf-8 /EHsc /bigobj)
endif()
```
`reproduce.ps1`:
```powershell
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
param(
[Parameter(Mandatory = $true)] [string] $TsFileSource,
[string] $BuildRoot = (Join-Path $PSScriptRoot 'build'),
[string] $JavaClasspath = '',
[string] $StaticLibrary = ''
)
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $false
function Invoke-Checked($Program, [string[]] $Arguments) {
& $Program @Arguments
if ($LASTEXITCODE -ne 0) { throw "$Program exited with $LASTEXITCODE" }
}
function Get-Value($Program, [string[]] $Arguments) {
$captured = & $Program @Arguments
if ($LASTEXITCODE -ne 0) { throw "$Program exited with $LASTEXITCODE" }
$captured | ForEach-Object { Write-Host $_ }
$values = @($captured | Select-String 'output.value=([0-9.]+)')
if ($values.Count -ne 1) { throw 'Expected exactly one output value' }
return $values[0].Matches[0].Groups[1].Value
}
$sourcePath = (Resolve-Path -LiteralPath $TsFileSource).Path
New-Item -ItemType Directory -Force -Path $BuildRoot | Out-Null
$BuildRoot = (Resolve-Path -LiteralPath $BuildRoot).Path
$java = if ($env:JAVA_HOME) { Join-Path $env:JAVA_HOME 'bin\java.exe' } else { 'java' }
$javac = if ($env:JAVA_HOME) { Join-Path $env:JAVA_HOME 'bin\javac.exe' } else { 'javac' }
if (-not $StaticLibrary) {
$libraryBuild = Join-Path $BuildRoot 'library'
Invoke-Checked cmake @('-S', "$sourcePath/cpp", '-B', $libraryBuild,
'-DTSFILE_BUILD_SHARED=OFF', '-DBUILD_TEST=OFF', '-DBUILD_TOOLS=OFF',
'-DENABLE_ANTLR4=OFF', '-DENABLE_SNAPPY=OFF', '-DENABLE_LZ4=OFF',
'-DENABLE_LZOKAY=OFF', '-DENABLE_ZLIB=OFF', '-DENABLE_ZSTD=OFF',
'-DENABLE_LZMA2=OFF', '-DENABLE_SIMD=OFF', '-DENABLE_THREADS=ON')
Invoke-Checked cmake @('--build', $libraryBuild, '--config', 'Release', '--target', 'tsfile', '--parallel', '6')
$StaticLibrary = Join-Path $libraryBuild 'lib\Release\tsfile.lib'
}
$programBuild = Join-Path $BuildRoot 'program'
Invoke-Checked cmake @('-S', $PSScriptRoot, '-B', $programBuild, "-DTSFILE_SOURCE=$sourcePath", "-DTSFILE_LIBRARY=$StaticLibrary")
Invoke-Checked cmake @('--build', $programBuild, '--config', 'Release', '--target', 'precision_repro')
if (-not $JavaClasspath) {
Invoke-Checked mvn @('-f', "$sourcePath/pom.xml", '-P', 'with-java', '-pl', 'java/tsfile', '-am',
'-Dtest=CompatibilityTest', '-Dsurefire.failIfNoSpecifiedTests=false', '-Dspotless.skip=true', 'test')
[xml] $report = Get-Content -LiteralPath "$sourcePath/java/tsfile/target/surefire-reports/TEST-org.apache.tsfile.compatibility.CompatibilityTest.xml" -Raw
$JavaClasspath = ($report.testsuite.properties.property | Where-Object { $_.name -eq 'java.class.path' }).value
if (-not $JavaClasspath) { throw 'Missing Java classpath' }
}
$classes = Join-Path $BuildRoot 'classes'
New-Item -ItemType Directory -Force -Path $classes | Out-Null
Invoke-Checked $javac @('-cp', $JavaClasspath, '-d', $classes, "$PSScriptRoot/PrecisionRepro.java")
$classpath = $JavaClasspath + ';' + $classes
$cpp = Join-Path $programBuild 'Release\precision_repro.exe'
$mismatches = 0
$controls = 0
foreach ($type in @('FLOAT', 'DOUBLE')) {
foreach ($alignment in @('nonaligned', 'aligned')) {
foreach ($precision in @('0', '2', '4', 'default')) {
$id = "$type-$alignment-$precision"
$javaFile = Join-Path $BuildRoot "$id-java.tsfile"
$cppFile = Join-Path $BuildRoot "$id-cpp.tsfile"
if ((Test-Path -LiteralPath $javaFile) -or (Test-Path -LiteralPath $cppFile)) { throw 'Use a fresh BuildRoot for each run' }
Write-Host "CASE=$id"
$javaValue = Get-Value $java @('-cp', $classpath, 'PrecisionRepro', 'write', $javaFile, $type, $precision, $alignment)
$cppValue = Get-Value $cpp @('write', $cppFile, $type, $precision, $alignment)
$javaReadsCpp = Get-Value $java @('-cp', $classpath, 'PrecisionRepro', 'read', $cppFile, $type, $precision, $alignment)
$cppReadsJava = Get-Value $cpp @('read', $javaFile, $type, $precision, $alignment)
$expected = switch ($precision) { '0' { '1.000000' }; '4' { '1.234600' }; default { '1.230000' } }
if ($javaValue -ne $expected -or $cppValue -ne '1.230000') { throw "Unexpected writer value: $id" }
if ($javaReadsCpp -ne $cppValue -or $cppReadsJava -ne $javaValue) { throw "Cross-read changed the stored value: $id" }
if ($javaValue -eq $cppValue) { $controls++ } else { $mismatches++ }
Write-Host "comparison.java=$javaValue comparison.cpp=$cppValue cross_read=PRESERVES_STORED_VALUE"
}
}
}
Write-Host "configurations=16 reproduced_mismatches=$mismatches matching_controls=$controls cross_reads=32"
if ($mismatches -ne 8 -or $controls -ne 8) { throw 'Unexpected reproduction summary' }
Write-Host 'reproduction=CONFIRMED'
```
### Cause and expected behavior
Java's [MeasurementSchema.getValueEncoder()](https://github.com/apache/tsfile/blob/12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f/java/tsfile/src/main/java/org/apache/tsfile/write/schema/MeasurementSchema.java#L279) initializes the encoding builder from the measurement properties. Its `Ts2Diff.initFromProps()` reads `max_point_number` and passes the selected precision to the floating-point encoder.
C++ [`TsFileWriter::do_check_schema()`](https://github.com/apache/tsfile/blob/12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f/cpp/src/writer/tsfile_writer.cc#L536) creates chunk/page encoders with the measurement name, data type, encoding and compression; `props_` is not passed down. Aligned value-page creation has the same missing property input. The floating-point encoders default to 2 and already expose [`FloatTS2DIFFEncoder::set_max_point_number()`](https://github.com/apache/tsfile/blob/12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f/cpp/src/encoding/ts2diff_encoder.h#L617) and its DOUBLE equivalent, but the normal writer paths do not call these setters. PR #901 explicitly described that remaining limitation; this issue covers the property wiring rather than that PR's page-format fix.
C++ should preserve measurement properties during registration and apply `max_point_number` when constructing FLOAT/DOUBLE TS_2DIFF encoders in both aligned and nonaligned writer paths. For the input above, both languages should store/read `1.000000`; with precision 4, both should store/read `1.234600`. Property-absent and explicit precision-2 behavior should remain consistent. The implementation should retain the configured precision across page resets and chunk flushes, and test Java's handling of malformed/negative settings separately.
Scope: FLOAT/DOUBLE TS_2DIFF writer property configuration. This does not require changing the existing decoder or adding floating-point RLE support.
### Evidence
The screenshot shows actual captured stdout in a browser log viewer, rather than a terminal capture. Raw logs, the four files and all 32 generated TsFiles are included in the accompanying reproduction package.
[Download the reproduction package](https://raw.githubusercontent.com/jixuan1989/tsfile-apache/8daac7037d40c93f4c1df4258095d3641dad2344/issue-evidence/ts2diff-writer-precision/reproduction.zip)
## 中文翻译
### 问题及相同输入
相同的 TS_2DIFF 测点属性和数据，Java writer 会应用 `max_point_number`，C++ writer 则继续使用两位小数。两边写入均成功，文件也均可读取，但实际存储值不同。因此，设置精度为 4 时，C++ 会丢失 Java 能够保留的精度。
```json
{
"model": "TREE",
"device": "root.precision.d",
"measurement": "v",
"type": "FLOAT",
"encoding": "TS_2DIFF",
"compression": "UNCOMPRESSED",
"properties": {"max_point_number": "0"},
"timestamp": 1,
"value": 1.23456
}
```
### 实际输入与输出
Java 写入并读取自己的文件：
```text
language=Java mode=write type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.000000
read=SUCCESS rows=1
```
当前 C++ 写入并读取自己的文件：
```text
language=C++ mode=write type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.230000
read=SUCCESS rows=1
```
两个进程的退出码均为 `0`。问题是存储值不同，并非读取失败。C++ 检查的各项操作均返回 `E_OK`；Java writer 的返回值表示是否达到刷写阈值，并非 `E_OK` 状态码。
Java 读取 C++ 文件：
```text
language=Java mode=read type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.230000
read=SUCCESS rows=1
```
C++ 读取 Java 文件：
```text
language=C++ mode=read type=FLOAT precision=0 alignment=nonaligned
input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED
output.timestamp=1 output.value=1.000000
read=SUCCESS rows=1
```
交叉读取保留了各自 writer 写入的值。差异发生在写入时的精度配置，两边的解码器均可处理这些文件。
### 补充案例及对照
| Type / alignment | max_point_number | Java writes → either reader | C++ writes → either reader |
| --- | --- | --- | --- |
| FLOAT and DOUBLE; nonaligned and aligned | 0 | 1.000000 | 1.230000 |
| FLOAT and DOUBLE; nonaligned and aligned | 4 | 1.234600 | 1.230000 |
| FLOAT and DOUBLE; nonaligned and aligned | 2 | 1.230000 | 1.230000 |
| FLOAT and DOUBLE; nonaligned and aligned | default (property absent) | 1.230000 | 1.230000 |
脚本检查 16 组 writer 配置：两种类型 × 两种对齐方式 × 四种精度设置，确认 8 组差异、8 组一致的对照以及 32 次成功交叉读取。上表保留六位小数，比较的是解码后的值，并未声称整个文件的字节完全相同。
### 复现环境与命令
两种实现均从上游提交 `12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f`（`develop`、`2.5.1-SNAPSHOT`）干净构建。实测环境为 Windows x64、MSVC 14.50、Release 静态 C++ 库、JDK 24.0.2。需要 JDK 17+、Maven、CMake 和 Visual Studio C++ 构建工具。如果 PATH 中没有 Java，将 `JAVA_HOME` 设置为 JDK 目录。
将下方四份完整文件保存至同一个目录，在该目录通过 PowerShell 对该提交的干净源码运行：
```powershell
.\reproduce.ps1 -TsFileSource D:\code\tsfile -BuildRoot .\build-precision
```
每次运行使用新的 `BuildRoot`。只有全部案例符合本文报告的行为，脚本才以 `0` 退出，表示缺陷已复现；最后一行应为 `reproduction=CONFIRMED`。可选压缩库关闭，时间列和值列均不压缩。C++ 树模型复现程序关闭了自动生成表 Schema，以排除 #1017 的干扰；注册测点时使用 Schema 指针向量重载，确保调用者的属性保留在已注册 Schema 中。
### 完整复现源码
保存为 `PrecisionRepro.java`，代码保持原样：
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
* either express or implied. See the License for the specific
* language governing permissions and limitations under the License.
*/
import org.apache.tsfile.common.conf.TSFileDescriptor;
import org.apache.tsfile.enums.TSDataType;
import org.apache.tsfile.file.metadata.enums.CompressionType;
import org.apache.tsfile.file.metadata.enums.TSEncoding;
import org.apache.tsfile.read.TsFileReader;
import org.apache.tsfile.read.TsFileSequenceReader;
import org.apache.tsfile.read.common.Path;
import org.apache.tsfile.read.common.RowRecord;
import org.apache.tsfile.read.expression.QueryExpression;
import org.apache.tsfile.read.query.dataset.QueryDataSet;
import org.apache.tsfile.write.TsFileWriter;
import org.apache.tsfile.write.record.TSRecord;
import org.apache.tsfile.write.schema.MeasurementSchema;
import java.io.File;
import java.util.List;
import java.util.Locale;
import java.util.Map;
public class PrecisionRepro {
public static void main(String[] args) throws Exception {
if (args.length != 5) throw new IllegalArgumentException("write|read file FLOAT|DOUBLE default|precision aligned|nonaligned");
TSDataType type = TSDataType.valueOf(args[2]);
boolean aligned = args[4].equals("aligned");
File file = new File(args[1]);
TSFileDescriptor.getInstance().getConfig().setCompressor("UNCOMPRESSED");
TSFileDescriptor.getInstance().getConfig().setFloatPrecision(2);
if (args[0].equals("write")) {
Map<String, String> props = args[3].equals("default") ? Map.of() : Map.of("max_point_number", args[3]);
MeasurementSchema schema = new MeasurementSchema("v", type, TSEncoding.TS_2DIFF, CompressionType.UNCOMPRESSED, props);
try (TsFileWriter writer = new TsFileWriter(file)) {
if (aligned) writer.registerAlignedTimeseries(new Path("root.precision.d"), List.of(schema));
else writer.registerTimeseries(new Path("root.precision.d"), schema);
TSRecord record = new TSRecord("root.precision.d", 1L);
if (type == TSDataType.FLOAT) record.addPoint("v", 1.23456f);
else record.addPoint("v", 1.23456d);
writer.writeRecord(record);
}
}
System.out.printf("language=Java mode=%s type=%s precision=%s alignment=%s%n", args[0], type, args[3], args[4]);
System.out.println("input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED");
try (TsFileReader reader = new TsFileReader(new TsFileSequenceReader(file.getPath()))) {
QueryDataSet result = reader.query(QueryExpression.create(List.of(new Path("root.precision.d", "v", true)), null));
int rows = 0;
while (result.hasNext()) {
RowRecord row = result.next();
double value = type == TSDataType.FLOAT ? row.getFields().get(0).getFloatV() : row.getFields().get(0).getDoubleV();
System.out.printf(Locale.ROOT, "output.timestamp=%d output.value=%.6f%n", row.getTimestamp(), value);
if (row.getTimestamp() != 1) throw new AssertionError("timestamp");
rows++;
}
if (rows != 1) throw new AssertionError("row count=" + rows);
System.out.println("read=SUCCESS rows=" + rows);
}
}
}
```
保存为 `precision_repro.cc`，代码保持原样：
```cpp
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
* either express or implied. See the License for the specific
* language governing permissions and limitations under the License.
*/
#include <iomanip>
#include <iostream>
#include <stdexcept>
#include <string>
#include <vector>
#include "reader/tsfile_reader.h"
#include "writer/tsfile_writer.h"
void check(int code, const char* operation) {
if (code != common::E_OK)
throw std::runtime_error(std::string(operation) + " code=" + std::to_string(code));
}
int run(int argc, char** argv) {
if (argc != 6) throw std::runtime_error("write|read file FLOAT|DOUBLE default|precision aligned|nonaligned");
const bool is_float = std::string(argv[3]) == "FLOAT";
const bool aligned = std::string(argv[5]) == "aligned";
const auto type = is_float ? common::FLOAT : common::DOUBLE;
common::g_config_value_.time_compress_type_ = common::UNCOMPRESSED;
if (std::string(argv[1]) == "write") {
storage::TsFileWriter writer;
check(writer.open(argv[2]), "open writer");
writer.set_generate_table_schema(false);
auto* schema = new storage::MeasurementSchema("v", type, common::TS_2DIFF, common::UNCOMPRESSED);
if (std::string(argv[4]) != "default") schema->props_["max_point_number"] = argv[4];
std::vector<storage::MeasurementSchema*> schemas{schema};
if (aligned) check(writer.register_aligned_timeseries("root.precision.d", schemas), "register aligned");
else check(writer.register_timeseries("root.precision.d", schemas), "register nonaligned");
storage::TsRecord record(1, "root.precision.d");
if (is_float) check(record.add_point("v", 1.23456f), "add float");
else check(record.add_point("v", 1.23456), "add double");
check(aligned ? writer.write_record_aligned(record) : writer.write_record(record), "write record");
check(writer.close(), "close writer");
}
std::cout << "language=C++ mode=" << argv[1] << " type=" << argv[3]
<< " precision=" << argv[4] << " alignment=" << argv[5] << '\n'
<< "input.timestamp=1 input.value=1.234560 encoding=TS_2DIFF compression=UNCOMPRESSED\n";
storage::TsFileReader reader;
check(reader.open(argv[2]), "open reader");
storage::ResultSet* result = nullptr;
std::vector<std::string> paths{"root.precision.d.v"};
check(reader.query(paths, 0, 2, result), "query");
int rows = 0;
for (;;) {
bool has_next = false;
check(result->next(has_next), "next");
if (!has_next) break;
double value = is_float ? result->get_value<float>(2) : result->get_value<double>(2);
int64_t timestamp = result->get_value<int64_t>(1);
std::cout << "output.timestamp=" << timestamp << " output.value=" << std::fixed << std::setprecision(6) << value << '\n';
if (timestamp != 1) throw std::runtime_error("timestamp");
++rows;
}
reader.destroy_query_data_set(result);
check(reader.close(), "close reader");
if (rows != 1) throw std::runtime_error("row count=" + std::to_string(rows));
std::cout << "read=SUCCESS rows=" << rows << '\n';
return 0;
}
int main(int argc, char** argv) {
try {
check(storage::libtsfile_init(), "initialize");
int code = run(argc, argv);
storage::libtsfile_destroy();
return code;
} catch (const std::exception& error) {
std::cerr << error.what() << '\n';
return 2;
}
}
```
### 完整构建与运行文件
将 `CMakeLists.txt` 和 `reproduce.ps1` 与两个程序放在同一目录。可选参数 `StaticLibrary` 和 `JavaClasspath` 用于复用已有构建；省略这两个参数时，脚本从指定的源码目录构建两种实现。
`CMakeLists.txt`：
```cmake
cmake_minimum_required(VERSION 3.15)
project(tsfile_precision_repro LANGUAGES CXX)
set(TSFILE_SOURCE "" CACHE PATH "Apache TsFile source checkout")
set(TSFILE_LIBRARY "" CACHE FILEPATH "Release static TsFile library")
set(TSFILE_BASELINE_INCLUDE "" CACHE PATH "Optional preserved baseline headers")
add_executable(precision_repro precision_repro.cc)
target_compile_features(precision_repro PRIVATE cxx_std_14)
target_compile_definitions(precision_repro PRIVATE TSFILE_STATIC NOMINMAX ENABLE_MEM_STAT ENABLE_THREADS)
target_include_directories(precision_repro PRIVATE "${TSFILE_SOURCE}/cpp/src")
if(TSFILE_BASELINE_INCLUDE)
target_include_directories(precision_repro BEFORE PRIVATE "${TSFILE_BASELINE_INCLUDE}")
endif()
target_link_libraries(precision_repro PRIVATE "${TSFILE_LIBRARY}")
if(MSVC)
target_compile_options(precision_repro PRIVATE /utf-8 /EHsc /bigobj)
endif()
```
`reproduce.ps1`：
```powershell
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
param(
[Parameter(Mandatory = $true)] [string] $TsFileSource,
[string] $BuildRoot = (Join-Path $PSScriptRoot 'build'),
[string] $JavaClasspath = '',
[string] $StaticLibrary = ''
)
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $false
function Invoke-Checked($Program, [string[]] $Arguments) {
& $Program @Arguments
if ($LASTEXITCODE -ne 0) { throw "$Program exited with $LASTEXITCODE" }
}
function Get-Value($Program, [string[]] $Arguments) {
$captured = & $Program @Arguments
if ($LASTEXITCODE -ne 0) { throw "$Program exited with $LASTEXITCODE" }
$captured | ForEach-Object { Write-Host $_ }
$values = @($captured | Select-String 'output.value=([0-9.]+)')
if ($values.Count -ne 1) { throw 'Expected exactly one output value' }
return $values[0].Matches[0].Groups[1].Value
}
$sourcePath = (Resolve-Path -LiteralPath $TsFileSource).Path
New-Item -ItemType Directory -Force -Path $BuildRoot | Out-Null
$BuildRoot = (Resolve-Path -LiteralPath $BuildRoot).Path
$java = if ($env:JAVA_HOME) { Join-Path $env:JAVA_HOME 'bin\java.exe' } else { 'java' }
$javac = if ($env:JAVA_HOME) { Join-Path $env:JAVA_HOME 'bin\javac.exe' } else { 'javac' }
if (-not $StaticLibrary) {
$libraryBuild = Join-Path $BuildRoot 'library'
Invoke-Checked cmake @('-S', "$sourcePath/cpp", '-B', $libraryBuild,
'-DTSFILE_BUILD_SHARED=OFF', '-DBUILD_TEST=OFF', '-DBUILD_TOOLS=OFF',
'-DENABLE_ANTLR4=OFF', '-DENABLE_SNAPPY=OFF', '-DENABLE_LZ4=OFF',
'-DENABLE_LZOKAY=OFF', '-DENABLE_ZLIB=OFF', '-DENABLE_ZSTD=OFF',
'-DENABLE_LZMA2=OFF', '-DENABLE_SIMD=OFF', '-DENABLE_THREADS=ON')
Invoke-Checked cmake @('--build', $libraryBuild, '--config', 'Release', '--target', 'tsfile', '--parallel', '6')
$StaticLibrary = Join-Path $libraryBuild 'lib\Release\tsfile.lib'
}
$programBuild = Join-Path $BuildRoot 'program'
Invoke-Checked cmake @('-S', $PSScriptRoot, '-B', $programBuild, "-DTSFILE_SOURCE=$sourcePath", "-DTSFILE_LIBRARY=$StaticLibrary")
Invoke-Checked cmake @('--build', $programBuild, '--config', 'Release', '--target', 'precision_repro')
if (-not $JavaClasspath) {
Invoke-Checked mvn @('-f', "$sourcePath/pom.xml", '-P', 'with-java', '-pl', 'java/tsfile', '-am',
'-Dtest=CompatibilityTest', '-Dsurefire.failIfNoSpecifiedTests=false', '-Dspotless.skip=true', 'test')
[xml] $report = Get-Content -LiteralPath "$sourcePath/java/tsfile/target/surefire-reports/TEST-org.apache.tsfile.compatibility.CompatibilityTest.xml" -Raw
$JavaClasspath = ($report.testsuite.properties.property | Where-Object { $_.name -eq 'java.class.path' }).value
if (-not $JavaClasspath) { throw 'Missing Java classpath' }
}
$classes = Join-Path $BuildRoot 'classes'
New-Item -ItemType Directory -Force -Path $classes | Out-Null
Invoke-Checked $javac @('-cp', $JavaClasspath, '-d', $classes, "$PSScriptRoot/PrecisionRepro.java")
$classpath = $JavaClasspath + ';' + $classes
$cpp = Join-Path $programBuild 'Release\precision_repro.exe'
$mismatches = 0
$controls = 0
foreach ($type in @('FLOAT', 'DOUBLE')) {
foreach ($alignment in @('nonaligned', 'aligned')) {
foreach ($precision in @('0', '2', '4', 'default')) {
$id = "$type-$alignment-$precision"
$javaFile = Join-Path $BuildRoot "$id-java.tsfile"
$cppFile = Join-Path $BuildRoot "$id-cpp.tsfile"
if ((Test-Path -LiteralPath $javaFile) -or (Test-Path -LiteralPath $cppFile)) { throw 'Use a fresh BuildRoot for each run' }
Write-Host "CASE=$id"
$javaValue = Get-Value $java @('-cp', $classpath, 'PrecisionRepro', 'write', $javaFile, $type, $precision, $alignment)
$cppValue = Get-Value $cpp @('write', $cppFile, $type, $precision, $alignment)
$javaReadsCpp = Get-Value $java @('-cp', $classpath, 'PrecisionRepro', 'read', $cppFile, $type, $precision, $alignment)
$cppReadsJava = Get-Value $cpp @('read', $javaFile, $type, $precision, $alignment)
$expected = switch ($precision) { '0' { '1.000000' }; '4' { '1.234600' }; default { '1.230000' } }
if ($javaValue -ne $expected -or $cppValue -ne '1.230000') { throw "Unexpected writer value: $id" }
if ($javaReadsCpp -ne $cppValue -or $cppReadsJava -ne $javaValue) { throw "Cross-read changed the stored value: $id" }
if ($javaValue -eq $cppValue) { $controls++ } else { $mismatches++ }
Write-Host "comparison.java=$javaValue comparison.cpp=$cppValue cross_read=PRESERVES_STORED_VALUE"
}
}
}
Write-Host "configurations=16 reproduced_mismatches=$mismatches matching_controls=$controls cross_reads=32"
if ($mismatches -ne 8 -or $controls -ne 8) { throw 'Unexpected reproduction summary' }
Write-Host 'reproduction=CONFIRMED'
```
### 原因与预期行为
Java 的 [MeasurementSchema.getValueEncoder()](https://github.com/apache/tsfile/blob/12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f/java/tsfile/src/main/java/org/apache/tsfile/write/schema/MeasurementSchema.java#L279) 用测点属性初始化编码构造器，`Ts2Diff.initFromProps()` 读取 `max_point_number`，再把选定精度传给浮点编码器。
C++ 的 [`TsFileWriter::do_check_schema()`](https://github.com/apache/tsfile/blob/12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f/cpp/src/writer/tsfile_writer.cc#L536) 创建 Chunk/Page 编码器时，仅传入测点名、数据类型、编码和压缩方式，没有向下传递 `props_`。对齐值页的创建路径也缺少属性输入。浮点编码器默认精度为 2，已有 [`FloatTS2DIFFEncoder::set_max_point_number()`](https://github.com/apache/tsfile/blob/12e39fe9d9dace63cc0ad39664aa0fbf5e29f52f/cpp/src/encoding/ts2diff_encoder.h#L617) 及 DOUBLE 对应接口，但正常 writer 路径没有调用这些 setter。PR #901 已明确说明这个遗留限制；本 issue 处理属性接线，与该 PR 的页格式修复范围不同。
C++ 应在测点注册时保留属性，并在对齐、非对齐 writer 路径构造 FLOAT/DOUBLE TS_2DIFF 编码器时应用 `max_point_number`。对于上面的输入，两种语言均应存储并读出 `1.000000`；精度为 4 时均应读出 `1.234600`。未设置属性及显式设置为 2 的行为应继续保持一致。实现应在 Page 重置、Chunk 刷写后保留已配置精度，并另行测试 Java 对格式错误和负数设置的处理。
范围：FLOAT/DOUBLE TS_2DIFF writer 的属性配置。本项无需改变已有解码器，也不包含浮点 RLE 支持。
### 证据
截图来自展示实际捕获 stdout 的浏览器日志预览器，不冒充终端截图。附带复现包包含原始日志、上述四份文件以及生成的全部 32 个 TsFile。
[下载复现包](https://raw.githubusercontent.com/jixuan1989/tsfile-apache/8daac7037d40c93f4c1df4258095d3641dad2344/issue-evidence/ts2diff-writer-precision/reproduction.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tsfile.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org