You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "vera-rubin (via GitHub)" <gi...@apache.org> on 2026/10/12 13:36:47 UTC
[I] [C++] FLOAT statistics use values before binary32 narrowing [orc]
vera-rubin opened a new issue, #2719:
URL: https://github.com/apache/orc/issues/2719
The default C++ writer stores FLOAT statistics from the double input before narrowing it to binary32. A one-row file containing `0.1` returns no rows for an equality search argument whose literal exactly equals the stored FLOAT value.
## Version and build
Apache ORC `3.0.0-SNAPSHOT`, commit `2069271e06c92197c66def9d4a0214753ac9a3b8`. Tested with GCC 13.3.0, CMake 3.28.3 and Python 3.12.3 on Ubuntu, Linux 6.17.0-1020-oracle, x86_64. C++ Release build, default numeric vectors and ZSTD compression. CMake Java and C++ tests were off. As checked on 2026-10-12, upstream `main` matches this commit.
## Reproduction
Save this as `repro.cc`, then run the Bash commands below from the same directory. Reproducer: CC0-1.0.
```cpp
#include <orc/OrcFile.hh>
#include <orc/Statistics.hh>
#include <orc/Vector.hh>
#include <orc/sargs/SearchArgument.hh>
#include <iostream>
#include <iomanip>
int main(int argc,char**argv){
if(argc!=2)return 2;
auto schema=orc::Type::buildTypeFromString("struct<x:float>");
{
auto out=orc::writeLocalFile(argv[1]);
auto writer=orc::createWriter(*schema,out.get(),orc::WriterOptions());
auto batch=writer->createRowBatch(1);
auto& root=dynamic_cast<orc::StructVectorBatch&>(*batch);
auto& x=dynamic_cast<orc::DoubleVectorBatch&>(*root.fields[0]);
root.numElements=x.numElements=1;x.data[0]=0.1;
writer->add(*batch);writer->close();
}
auto reader=orc::createReader(orc::readLocalFile(argv[1]),orc::ReaderOptions());
auto stats=reader->getColumnStatistics(1);
auto& f=dynamic_cast<orc::DoubleColumnStatistics&>(*stats);
std::cout<<std::setprecision(17)<<"minimum "<<f.getMinimum()<<"\nmaximum "<<f.getMaximum()<<"\nsum "<<f.getSum()<<"\n";
for(bool filtered:{false,true}){
orc::RowReaderOptions opts;
if(filtered) opts.searchArgument(orc::SearchArgumentFactory::newBuilder()->startAnd()
.equals("x",orc::PredicateDataType::FLOAT,orc::Literal(0.100000001490116119384765625)).end().build());
auto rows=reader->createRowReader(opts);auto batch=rows->createRowBatch(1);uint64_t count=0;
while(rows->next(*batch)){
auto& root=dynamic_cast<orc::StructVectorBatch&>(*batch);
auto& x=dynamic_cast<orc::DoubleVectorBatch&>(*root.fields[0]);
for(uint64_t i=0;i<root.numElements;i++) std::cout<<(filtered?"filtered_value ":"plain_value ")<<x.data[i]<<"\n";
count+=root.numElements;
}
std::cout<<(filtered?"filtered_rows ":"plain_rows ")<<count<<"\n";
}
}
```
```sh
git clone https://github.com/apache/orc.git orc
git -C orc checkout --detach 2069271e06c92197c66def9d4a0214753ac9a3b8
cmake -S orc -B orc-build \
-DBUILD_JAVA=OFF -DBUILD_CPP_TESTS=OFF -DBUILD_TOOLS=ON \
-DCMAKE_BUILD_TYPE=Release -DSTOP_BUILD_ON_WARNING=OFF
cmake --build orc-build --parallel 2
g++ -std=c++17 -O2 -Iorc/c++/include -Iorc-build/c++/include \
repro.cc orc-build/c++/src/liborc.a \
orc-build/_deps/protobuf-build/liborc_vendored_protobuf.a \
orc-build/_deps/zlib-build/liborc_vendored_zlib.a \
orc-build/_deps/snappy-build/liborc_vendored_snappy.a \
orc-build/_deps/lz4-build/liborc_vendored_lz4.a \
orc-build/_deps/zstd-build/lib/liborc_vendored_zstd.a \
-pthread -o repro
./repro input.orc
```
## Expected and observed
The [column statistics specification](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/site/specification/ORCv1.md#L260-L265) defines statistics over column values. [FLOAT encoding](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/site/specification/ORCv1.md#L1051-L1055) uses IEEE 754 binary32. The [C++ vector documentation](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/site/_docs/core-cpp.md#L40-L52) represents FLOAT input with `DoubleVectorBatch`.
The stored value has bits `0x3dcccccd`. Its exact value is `13421773/134217728`, or `0.100000001490116119384765625`. With one non-null row, minimum, maximum and sum should all equal that value. The equality search argument should retain the row.
Observed:
```text
minimum 0.10000000000000001
maximum 0.10000000000000001
sum 0.10000000000000001
plain_value 0.10000000149011612
plain_rows 1
filtered_rows 0
```
An independent byte decoder confirms the DATA value and the mismatching statistics at file, stripe and row-group levels. The literal equals the decoded FLOAT exactly.
## Source and scope
In [FloatingColumnWriter::add](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/c%2B%2B/src/ColumnWriter.cc#L869-L881), DATA uses `static_cast<float>(doubleData[i])`, while statistics use the original double. Statistics should use the value serialized as FLOAT.
Independent C++ controls pass with pre-rounded input, tight float vectors, exact `0.5`, and DOUBLE. Negative `-0.1` also reproduces. No read-type conversion is used. The independent harness initializes row counts and null flags and closes the writer before reading. Predicate pushdown may retain extra rows, but it must retain this matching row.
Java's FLOAT writer was compared with source at the same commit only. No Java writer runtime comparison or other compiler or architecture test is claimed.
Found by ChipLab's AI verification agents and independently reproduced.
ChipLab ref: CLAB-2026-863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org