You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2026/10/08 03:24:06 UTC
(doris) 35/35: [fix](arrow) Correct year-zero DATE conversion on branch-4.1 (#68649)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0e741f9162bd03cdca1822ee3f9c1e3bc31aa91f
Author: Gabriel <li...@selectdb.com>
AuthorDate: Thu Oct 8 10:48:32 2026 +0800
[fix](arrow) Correct year-zero DATE conversion on branch-4.1 (#68649)
### What problem does this PR solve?
Related issue: #67366
Related PR: #67449
On branch-4.1, Arrow Flight SQL returns `0000-01-02` for a stored DATE
of `0000-01-01`. Doris follows the MySQL calendar, which excludes year
zero's leap day, while Arrow DATE uses the proleptic Gregorian calendar.
Subtracting the Unix epoch day number directly shifts dates in
`0000-01-01` through `0000-02-28` by one day, including elements of
`ARRAY<DATE>`.
Adapt the DATE portion of #67449 to branch-4.1, with credit to its
author. Convert between the calendars at Arrow and ORC write boundaries
and Arrow/default Parquet/ORC read boundaries. Use the same conversion
for ORC statistics so pushed-down MIN/MAX agrees with row decoding.
Reject unrepresentable external dates and ignore undefined payloads in
null Arrow slots.
The two DATE regression suites and their expected results are copied
unchanged from #67449. Add its DATE unit coverage plus an explicit
null-payload test and an exhaustive comparison against the civil
calendar for all 3,652,424 representable dates.
Also align Iceberg DATE day/bucket partition transforms with file
ordinals. For ORC/Parquet, metadata is usable only when its raw DATE
range proves that every ordinal is representable, including the interior
of the range. Unsafe ranges fall back to decoding: strict reads retain
their errors, non-strict Parquet reads retain conversion-generated
NULLs, and COUNT uses materialized values when physical null counts are
insufficient. ORC uses a metadata-only SARG to check projected DATE
leaves per stripe: safe excluded stripes retain SARG pruning, while
unsafe stripes remain readable with SDK row-group pruning disabled. ORC
COUNT validates every DATE leaf of its argument in each selected stripe,
and strict Parquet COUNT validates projected DATE descendants before
reading only levels.
Scope: DATE, including its Iceberg partition transforms. This does not
include DATETIME changes or the independent legacy file scanner
conversions used when `enable_file_scanner_v2=false`. V1 reader changes
and mixed-version/rolling-upgrade compatibility gates are explicitly
outside this PR's scope.
### Validation
- Baseline: 9 of the original 10 DATE unit tests fail; the unchanged
Flight regression reproduces the one-day shift.
- Fixed: 250 unit tests pass under ASAN, including the complete ORC
reader suite, DATE SerDe exhaustive calendar check, Iceberg partition
transforms, Parquet statistics, DATE scans and COUNT tests. These
compile the modified branch-4.1 sources against cached test
dependencies.
- The unchanged `test_date_year_zero` Flight regression passes locally,
including nullable and array values.
- `test_outfile_date_year_zero` passes for Parquet and ORC using the
original rows and golden assertions, with S3 transport replaced by local
outfile/TVF only for the local run. The committed suite retains S3
transport for CI.
- Local integration runs use the same fix rebuilt into an existing
ASAN/UBSAN BE build at `5bcb996a5ff`, with an existing FE. A full
branch-4.1 build and the original S3 run remain for CI.
- clang-format 16 passes for all affected C++ files. Source whitespace
checks pass; generated golden files retain their conventional trailing
blank line from #67449.
- Review coverage: new tests reproduce partition mismatches,
invalid-interior MIN/MAX bypass, ORC SARG/Parquet footer/page pruning
inconsistencies, and Parquet IS NULL/COUNT discrepancies. All five
reproducing tests fail before the review fixes.
- Added a DATE-only Spark/Iceberg partitioned write/read regression,
including selective day/bucket predicates and partition metadata
comparison. Its Groovy compilation passes; execution requires the
external CI environment.
- Follow-up review: scalar/nested ORC COUNT and strict nested Parquet
COUNT reproductions fail before the guards and pass afterward. ARRAY,
STRUCT, MAP, all-null values, safe ranges, non-strict Parquet, and
unprojected DATE siblings are covered.
- External ORC regression: retain the original LZ4 file as a BE fixture
and verify that ordinal `-719530` raises a data-quality error. Replace
the obsolete `1900-01-01` golden result with an explicit error
assertion, restoring the scanner setting afterward. The fixture test and
Groovy compilation pass locally. The complete group6 suite is also
validated locally with HDFS transport replaced by LOCAL TVF; the
committed suite retains HDFS for CI.
- Pruning review: both new tests fail before the fixes and pass
afterward. Partial ORC STRUCT projection retains SARG when the invalid
DATE sibling is not read; selecting that DATE still preserves its error.
Multi-page Parquet tests retain the unsafe DATE page while pruning safe
pages for IS NULL and range predicates, through both single-slot and
multi-slot OR evaluators.
- Mixed-stripe ORC coverage: the original file-wide guard prunes zero
safe stripes; the fix prunes both safe excluded stripes and still
reports the invalid DATE in the retained stripe. The active SDK options
outlive their reader, and the cached SARG evaluator is discarded before
opening an unsafe range.
- External group6 coverage: the original `non_vec_orc_scanner.orc`
fixture encodes `20191111` as an invalid epoch-day ordinal. Replace its
obsolete `1900-01-01` golden result with an explicit data-quality error
assertion and keep the original bytes as a small BE fixture; projecting
other columns still succeeds.
- Final local integration after the stripe fix: all three suites pass
with no failures or skips (complete group6: 41 fixtures, 39
golden-result assertions and two expected errors; Flight year zero;
Parquet/ORC outfile year zero). These use the incremental ASAN/UBSAN
build and local transport substitutions described above.
- Metadata safety review: the malformed pruned-stripe index and
truncated nested DATE statistics reproduce SDK bounds assertions before
the fix. Both regressions pass afterward; the DATE preflight now adds no
file read calls or bytes compared with the same excluded stripe without
DATE projection.
- Historical-file migration: an explicit offline tool converts only
confirmed legacy Doris DATE ordinals, including nested values. It
publishes a separate output atomically and refuses overwrites. Tests
cover year-zero boundaries, NULLs, hidden null-container payloads,
nested ORC/Parquet round trips, schema metadata, timestamp precision,
and failed-write cleanup.
- Final follow-up validation: all 250 ASAN tests and 10 migration-tool
tests pass. The final incremental production BE passes the three
existing local suites plus six legacy migration/read/export/read checks,
covering both ORC and Parquet. Original HDFS/S3 transport and the full
branch build remain for CI as described above.
### Release note
Fix year-zero DATE values being returned one day late through Arrow
Flight SQL, align DATE encoding and decoding in the default Parquet/ORC
paths, and keep Iceberg DATE partition values consistent with file
contents.
Existing files written with the former year-zero encoding cannot be
distinguished automatically from correctly encoded files. Before
switching their consumers to the corrected reader, use the explicit
[legacy DATE migration tool and
procedure](https://github.com/apache/doris/blob/082429b00f2392f8e8eabcee6e6a5a94a403e7aa/tools/legacy_date_migration/README.md)
on files confirmed to use the old Doris encoding, or re-export from
authoritative original data. The tool writes new ORC/Parquet files
without modifying the source. Existing Iceberg files must be migrated
through a new table or a supported table-format transaction that
recomputes partition metadata; do not replace files in existing
snapshots. The proleptic-only `0000-02-29` remains unrepresentable as a
Doris DATE.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes, external DATE ordinals use the correct calendar; internal
DATE arithmetic is unchanged.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../data_type_serde/data_type_datev2_serde.cpp | 73 ++-
be/src/core/value/vdatetime_value.h | 53 +-
.../sink/writer/iceberg/partition_transformers.h | 8 +-
be/src/format_v2/orc/orc_reader.cpp | 164 +++++-
be/src/format_v2/parquet/parquet_reader.cpp | 43 ++
be/src/format_v2/parquet/parquet_statistics.cpp | 10 +
.../data_type_datev2_serde_calendar_test.cpp | 358 +++++++++++++
.../writer/iceberg/partition_transformers_test.cpp | 27 +
.../orc_scanner/date_count_truncated_stats.orc | Bin 0 -> 230 bytes
.../test_data/orc_scanner/date_large_ordinal.orc | Bin 0 -> 981 bytes
.../orc_scanner/date_metadata_fixtures.md | 39 ++
.../test_data/orc_scanner/date_out_of_range.orc | Bin 0 -> 619 bytes
.../exec/test_data/orc_scanner/date_preflight.orc | Bin 0 -> 390 bytes
.../orc_scanner/date_preflight_bad_index.orc | Bin 0 -> 390 bytes
be/test/format_v2/orc/orc_reader_test.cpp | 559 ++++++++++++++++++++-
be/test/format_v2/parquet/parquet_reader_test.cpp | 248 +++++++++
.../format_v2/parquet/parquet_statistics_test.cpp | 174 +++++++
.../arrow_flight_sql_p0/test_date_year_zero.out | 11 +
.../outfile/test_outfile_date_year_zero.out | 37 ++
.../tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out | 34 --
.../arrow_flight_sql_p0/test_date_year_zero.groovy | 95 ++++
.../outfile/test_outfile_date_year_zero.groovy | 85 ++++
.../write/test_iceberg_write_date_year_zero.groovy | 79 +++
.../orc_tvf/test_hdfs_orc_group6_orc_files.groovy | 38 +-
tools/legacy_date_migration/README.md | 117 +++++
.../legacy_date_migration/migrate_legacy_dates.py | 162 ++++++
.../test_migrate_legacy_dates.py | 179 +++++++
27 files changed, 2505 insertions(+), 88 deletions(-)
diff --git a/be/src/core/data_type_serde/data_type_datev2_serde.cpp b/be/src/core/data_type_serde/data_type_datev2_serde.cpp
index a4cc34f49fc..3b6f5f7fb17 100644
--- a/be/src/core/data_type_serde/data_type_datev2_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_datev2_serde.cpp
@@ -22,6 +22,7 @@
#include <fmt/core.h>
#include <cstdint>
+#include <limits>
#include <vector>
#include "common/config.h"
@@ -42,13 +43,9 @@
namespace doris {
#include "common/compile_check_begin.h"
-// This number represents the number of days from 0000-01-01 to 1970-01-01.
-static constexpr int32_t date_threshold = 719528;
namespace {
-constexpr int32_t DORIS_DATE_EPOCH_DAYNR = 719528;
-
Status decode_date_orc_values(const DataTypeSerDe& serde, IColumn& column,
const OrcDecodedColumnView& orc_view) {
const auto* orc_batch = dynamic_cast<const ::orc::LongVectorBatch*>(orc_view.batch);
@@ -65,22 +62,44 @@ Status decode_date_orc_values(const DataTypeSerDe& serde, IColumn& column,
orc_serde_utils::orc_decode_row_count(orc_view.rows, orc_view.selected_rows);
std::vector<int32_t> date_values;
date_values.resize(output_rows);
- auto& date_dict = date_day_offset_dict::get();
for (size_t row = 0; row < output_rows; ++row) {
const auto source_row = orc_serde_utils::orc_source_row_at(row, orc_view.selected_rows);
- const auto date = date_dict[cast_set<int>(orc_batch->data[source_row])];
- date_values[row] = cast_set<int32_t>(date.daynr() - DORIS_DATE_EPOCH_DAYNR);
+ // The payload of a null slot is undefined, so it is zeroed rather than range-checked;
+ // `read_column_from_decoded_values` never decodes it.
+ if (view.null_map != nullptr && view.null_map[row] != 0) {
+ date_values[row] = 0;
+ continue;
+ }
+ // ORC DATE is days since 1970-01-01 in the proleptic Gregorian calendar, the same encoding
+ // `decode_epoch_days()` expects, but ORC hands it over in an int64 batch. Reject a value
+ // that does not fit the int32 view here, while the real file value is still available for
+ // the message, instead of narrowing it blindly.
+ const int64_t file_days = orc_batch->data[source_row];
+ if (file_days < std::numeric_limits<int32_t>::min() ||
+ file_days > std::numeric_limits<int32_t>::max()) {
+ return Status::DataQualityError(
+ "DATE value {} is outside the Doris DATE range (0000-01-01..9999-12-31, "
+ "0000-02-29 excluded)",
+ file_days);
+ }
+ date_values[row] = static_cast<int32_t>(file_days);
}
view.values = reinterpret_cast<const uint8_t*>(date_values.data());
RETURN_IF_ERROR(orc_serde_utils::read_decoded_values(serde, column, &view));
return Status::OK();
}
-Status decode_parquet_date(int32_t encoded_date, DateV2Value<DateV2ValueType>* value) {
+// Shared by every "days since 1970-01-01" source: Parquet, ORC and Arrow date32/date64. The
+// message must therefore not name one format, and it carries the value so a broken file can be
+// identified without re-reading it.
+Status decode_epoch_days(int64_t encoded_date, DateV2Value<DateV2ValueType>* value) {
DORIS_CHECK(value != nullptr);
- const int64_t day_number = static_cast<int64_t>(encoded_date) + date_threshold;
- if (day_number < 0 || !value->get_date_from_daynr(static_cast<uint64_t>(day_number))) {
- return Status::DataQualityError("Parquet DATE value is out of range");
+ const int64_t day_number = epoch_days_to_daynr(encoded_date);
+ if (day_number == 0 || !value->get_date_from_daynr(static_cast<uint64_t>(day_number))) {
+ return Status::DataQualityError(
+ "DATE value {} is outside the Doris DATE range (0000-01-01..9999-12-31, "
+ "0000-02-29 excluded)",
+ encoded_date);
}
return Status::OK();
}
@@ -100,7 +119,7 @@ public:
_data.resize(old_size + num_values);
for (size_t row = 0; row < num_values; ++row) {
DateV2Value<DateV2ValueType> value;
- const auto status = decode_parquet_date(
+ const auto status = decode_epoch_days(
unaligned_load<int32_t>(values + row * sizeof(int32_t)), &value);
if (!status.ok()) {
if (_state != nullptr && _state->mark_conversion_failure(old_size + row)) {
@@ -217,13 +236,12 @@ Status DataTypeDateV2SerDe::write_column_to_arrow(const IColumn& column, const N
const auto& col_data = static_cast<const ColumnDateV2&>(column).get_data();
auto& date32_builder = assert_cast<arrow::Date32Builder&>(*array_builder);
for (size_t i = start; i < end; ++i) {
- auto daynr = col_data[i].daynr() - date_threshold;
if (null_map && (*null_map)[i]) {
RETURN_IF_ERROR(checkArrowStatus(date32_builder.AppendNull(), column, *array_builder));
} else {
- RETURN_IF_ERROR(
- checkArrowStatus(date32_builder.Append(cast_set<int, int64_t, false>(daynr)),
- column, *array_builder));
+ RETURN_IF_ERROR(checkArrowStatus(
+ date32_builder.Append(daynr_to_epoch_days(col_data[i].daynr())), column,
+ *array_builder));
}
}
return Status::OK();
@@ -258,10 +276,9 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
"Arrow Date64 value must contain whole days: row={}, milliseconds={}",
value_i, milliseconds);
}
- const int64_t daynr = milliseconds / MILLISECONDS_PER_DAY + date_threshold;
DateV2Value<DateV2ValueType> value;
- if (daynr <= 0 || daynr > DATE_MAX_DAYNR ||
- !value.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
+ if (const auto status = decode_epoch_days(milliseconds / MILLISECONDS_PER_DAY, &value);
+ !status.ok()) {
return Status::InvalidArgument(
"Arrow Date64 value is outside the Doris DATE range: "
"row={}, milliseconds={}",
@@ -278,12 +295,20 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
const auto* base_ptr = reinterpret_cast<const uint8_t*>(concrete_array->raw_values());
const size_t element_size = sizeof(int32_t);
for (auto value_i = start; value_i < end; ++value_i) {
- int32_t date_value = 0;
+ // A null slot has no calendar value and its payload may be outside the DATE range.
+ if (concrete_array->IsNull(value_i)) {
+ col_data.emplace_back(DateV2Value<DateV2ValueType>());
+ continue;
+ }
const uint8_t* raw_byte_ptr = base_ptr + value_i * element_size;
- memcpy(&date_value, raw_byte_ptr, element_size);
+ auto date_value = unaligned_load<int32_t>(raw_byte_ptr);
DateV2Value<DateV2ValueType> v;
- v.get_date_from_daynr(date_value + date_threshold);
+ if (const auto status = decode_epoch_days(date_value, &v); !status.ok()) {
+ return Status::InvalidArgument(
+ "Arrow Date32 value is outside the Doris DATE range: row={}, days={}",
+ value_i, date_value);
+ }
col_data.emplace_back(v);
}
} else {
@@ -311,7 +336,7 @@ Status DataTypeDateV2SerDe::read_column_from_decoded_values(IColumn& column,
continue;
}
DateV2Value<DateV2ValueType> date_v2;
- const auto status = decode_parquet_date(values[row], &date_v2);
+ const auto status = decode_epoch_days(values[row], &date_v2);
if (!status.ok()) {
// Decoded values back both metadata conversion and native materialization. Preserve
// strict errors while allowing nullable non-strict scans to mark only the bad row.
@@ -403,7 +428,7 @@ Status DataTypeDateV2SerDe::write_column_to_orc(const std::string& timezone, con
if (cur_batch->notNull[row_id] == 0) {
continue;
}
- cur_batch->data[row_id] = col_data[row_id].daynr() - date_threshold;
+ cur_batch->data[row_id] = daynr_to_epoch_days(col_data[row_id].daynr());
}
cur_batch->numElements = end - start;
return Status::OK();
diff --git a/be/src/core/value/vdatetime_value.h b/be/src/core/value/vdatetime_value.h
index 5b5f768c0e2..02fefdf3abf 100644
--- a/be/src/core/value/vdatetime_value.h
+++ b/be/src/core/value/vdatetime_value.h
@@ -1697,8 +1697,6 @@ private:
static constexpr int START_YEAR = 1900; // 1900-01-01
static constexpr int END_YEAR = 2039; // 2039-10-24
- static constexpr int DAY_OFFSET_CAL_START_POINT_DAYNR =
- 719528; // 1970-01-01 (start from 0000-01-01, 0000-01-01 is day 1, returns 1)
static std::array<DateV2Value<DateV2ValueType>, DICT_DAYS> DATE_DAY_OFFSET_ITEMS;
static std::array<std::array<std::array<int, 31>, 12>, 140> DATE_DAY_OFFSET_DICT;
@@ -1713,6 +1711,9 @@ private:
date_day_offset_dict& operator=(const date_day_offset_dict&) = default;
public:
+ static constexpr int DAY_OFFSET_CAL_START_POINT_DAYNR =
+ 719528; // 1970-01-01 (start from 0000-01-01, 0000-01-01 is day 1, returns 1)
+
static bool can_speed_up_calc_daynr(int year) { return year >= START_YEAR && year <= END_YEAR; }
static int get_offset_by_daynr(int daynr) { return daynr - DAY_OFFSET_CAL_START_POINT_DAYNR; }
@@ -1773,6 +1774,54 @@ inline uint32_t calc_daynr(uint16_t year, uint8_t month, uint8_t day) {
return delsum + y / 4 - y / 100 + y / 400;
}
+// Doris follows MySQL's calendar, in which year 0 is NOT a leap year (see `is_leap()`): 0000-02-29
+// does not exist and `calc_daynr()` therefore numbers 0000-01-01 .. 0000-02-28 one day ahead of the
+// proleptic Gregorian calendar. Arrow `date32`, Parquet/ORC `DATE` and the Iceberg spec all define
+// their day ordinal in the proleptic Gregorian calendar, where year 0 IS a leap year. The two
+// numberings coincide from 0000-03-01 (daynr 60) onwards, so the whole difference is the missing
+// 0000-02-29. Every conversion between a Doris DATE and an external "days since 1970-01-01" value
+// must go through the two helpers below instead of adding/subtracting the epoch daynr directly.
+inline constexpr int64_t DAYNR_OF_UNIX_EPOCH =
+ date_day_offset_dict::DAY_OFFSET_CAL_START_POINT_DAYNR;
+inline constexpr int64_t DAYNR_OF_0000_03_01 = 60; // first daynr shared by both calendars
+
+// Doris daynr -> days since 1970-01-01 in the proleptic Gregorian calendar.
+inline constexpr int32_t daynr_to_epoch_days(int64_t daynr) {
+ return static_cast<int32_t>(daynr - DAYNR_OF_UNIX_EPOCH -
+ (daynr < DAYNR_OF_0000_03_01 ? 1 : 0));
+}
+
+// The three boundaries of the external ordinal domain, derived from the daynr domain rather than
+// typed out, so a change to either calendar constant cannot leave them behind.
+inline constexpr int64_t EPOCH_DAYS_MIN = daynr_to_epoch_days(1); // 0000-01-01, smallest Doris DATE
+inline constexpr int64_t EPOCH_DAYS_0000_02_29 =
+ daynr_to_epoch_days(DAYNR_OF_0000_03_01) - 1; // exists in proleptic Gregorian only
+inline constexpr int64_t EPOCH_DAYS_MAX =
+ daynr_to_epoch_days(DATE_MAX_DAYNR); // 9999-12-31, the largest Doris DATE
+
+static_assert(DAYNR_OF_UNIX_EPOCH == 719528, "calc_daynr(1970, 1, 1)");
+static_assert(EPOCH_DAYS_MIN == -719528);
+static_assert(EPOCH_DAYS_0000_02_29 == -719469);
+static_assert(EPOCH_DAYS_MAX == 2932896);
+
+// Inverse of `daynr_to_epoch_days()`. Returns 0, which is never a valid daynr, when the value has
+// no Doris DATE representation: before 0000-01-01, after 9999-12-31, or the proleptic-only
+// 0000-02-29. Callers must reject 0 instead of feeding it to `get_date_from_daynr()`.
+inline constexpr int64_t epoch_days_to_daynr(int64_t epoch_days) {
+ if (epoch_days < EPOCH_DAYS_MIN || epoch_days > EPOCH_DAYS_MAX ||
+ epoch_days == EPOCH_DAYS_0000_02_29) {
+ return 0;
+ }
+ return epoch_days + DAYNR_OF_UNIX_EPOCH + (epoch_days < EPOCH_DAYS_0000_02_29 ? 1 : 0);
+}
+
+// Valid endpoints alone do not prove a DATE range is decodable: it may enclose the
+// Gregorian-only leap day. Metadata shortcuts require every ordinal in the range to be valid.
+inline constexpr bool epoch_days_range_is_representable(int64_t min, int64_t max) {
+ return min <= max && epoch_days_to_daynr(min) != 0 && epoch_days_to_daynr(max) != 0 &&
+ (max < EPOCH_DAYS_0000_02_29 || min > EPOCH_DAYS_0000_02_29);
+}
+
// DAY / WEEK fast path. Real workloads hold dates that cluster in a narrow range, so the two
// dictionary lookups below (`daynr(y, m, d)` and `daynr -> date`) are L1-resident; measured about
// 3x faster than the generic `date_add_interval<DAY>` (no TimeInterval, no second-level
diff --git a/be/src/exec/sink/writer/iceberg/partition_transformers.h b/be/src/exec/sink/writer/iceberg/partition_transformers.h
index 46db5ed2263..da63554ff03 100644
--- a/be/src/exec/sink/writer/iceberg/partition_transformers.h
+++ b/be/src/exec/sink/writer/iceberg/partition_transformers.h
@@ -658,7 +658,8 @@ public:
DateV2Value<DateV2ValueType> value =
binary_cast<uint32_t, DateV2Value<DateV2ValueType>>(*(UInt32*)p_in);
- int64_t days_from_unix_epoch = value.daynr() - 719528;
+ // Partition hashes must use the same Gregorian ordinal as the DATE file payload.
+ int64_t days_from_unix_epoch = daynr_to_epoch_days(value.daynr());
uint32_t hash_value = HashUtil::murmur_hash3_32(&days_from_unix_epoch,
sizeof(days_from_unix_epoch), 0);
@@ -1149,9 +1150,8 @@ public:
while (p_in < end_in) {
DateV2Value<DateV2ValueType> value =
binary_cast<uint32_t, DateV2Value<DateV2ValueType>>(*(UInt32*)p_in);
- // datetime_diff<DAY> actually returns int
- *p_out = cast_set<int, int64_t, false>(
- datetime_diff<DAY>(PartitionColumnTransformUtils::epoch_date(), value));
+ // Iceberg day partitions describe external DATE ordinals, not MySQL day numbers.
+ *p_out = daynr_to_epoch_days(value.daynr());
++p_in;
++p_out;
}
diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp
index 856f38f0eee..d87bad294c7 100644
--- a/be/src/format_v2/orc/orc_reader.cpp
+++ b/be/src/format_v2/orc/orc_reader.cpp
@@ -487,10 +487,30 @@ bool set_date_zone_map(const ::orc::ColumnStatistics& statistics, segment_v2::Zo
!date_statistics->hasMaximum()) {
return false;
}
- auto& date_dict = date_day_offset_dict::get();
- return set_validated_zone_map(
- Field::create_field<TYPE_DATEV2>(date_dict[date_statistics->getMinimum()]),
- Field::create_field<TYPE_DATEV2>(date_dict[date_statistics->getMaximum()]), zone_map);
+ if (!epoch_days_range_is_representable(date_statistics->getMinimum(),
+ date_statistics->getMaximum())) {
+ return false;
+ }
+ // ORC DATE statistics are proleptic-Gregorian day ordinals, the same domain the row decoder
+ // (DataTypeDateV2SerDe::read_column_from_orc) interprets. Converting them through
+ // `date_day_offset_dict` instead would put the year-zero window one day off the rows and let a
+ // pushed-down MIN/MAX report a value no row holds. A bound with no Doris DATE disables the
+ // statistics, so MIN/MAX falls back to scanning rows.
+ const auto to_date = [](int64_t epoch_days) -> std::optional<DateV2Value<DateV2ValueType>> {
+ const int64_t daynr = epoch_days_to_daynr(epoch_days);
+ DateV2Value<DateV2ValueType> value;
+ if (daynr == 0 || !value.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
+ return std::nullopt;
+ }
+ return value;
+ };
+ const auto min_value = to_date(date_statistics->getMinimum());
+ const auto max_value = to_date(date_statistics->getMaximum());
+ if (!min_value.has_value() || !max_value.has_value()) {
+ return false;
+ }
+ return set_validated_zone_map(Field::create_field<TYPE_DATEV2>(*min_value),
+ Field::create_field<TYPE_DATEV2>(*max_value), zone_map);
}
std::optional<DateV2Value<DateTimeV2ValueType>> datetime_v2_from_orc_millis(
@@ -740,6 +760,7 @@ struct OrcReaderScanState {
uint64_t last_stripe = 0;
uint64_t offset = 0;
uint64_t length = 0;
+ bool disable_sarg = false;
};
std::unique_ptr<::orc::Reader> reader;
@@ -747,6 +768,7 @@ struct OrcReaderScanState {
const ::orc::Type* root_type = nullptr;
::orc::ReaderMetrics reader_metrics;
::orc::RowReaderOptions row_reader_options; // projection + filter + SARG + stripe range
+ ::orc::RowReaderOptions active_row_reader_options;
std::string timezone = TimezoneUtils::default_time_zone;
cctz::time_zone timezone_obj;
std::unique_ptr<::orc::RowReader> row_reader;
@@ -771,6 +793,7 @@ struct OrcReaderScanState {
bool orc_lazy_selection_valid = false;
OrcSargBuildOptions sarg_build_options;
+ std::vector<uint64_t> decoded_date_column_ids;
std::vector<StripeRange> selected_stripe_ranges;
size_t current_stripe_range = 0;
bool stripe_pruning_applied = false;
@@ -1502,6 +1525,35 @@ Status OrcReader::_init_search_argument_from_local_filters() {
}
try {
+ const auto collect_dates = [&](const auto& self, const ::orc::Type& type,
+ const std::set<uint64_t>* projected_ids) -> void {
+ if (projected_ids != nullptr && !projected_ids->contains(type.getColumnId())) {
+ return;
+ }
+ if (type.getKind() == ::orc::TypeKind::DATE) {
+ _state->decoded_date_column_ids.push_back(type.getColumnId());
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ self(self, *type.getSubtype(child), projected_ids);
+ }
+ };
+ for (const auto column_id : _state->read_columns) {
+ if (is_virtual_column(column_id)) {
+ continue;
+ }
+ const auto& type =
+ *_state->root_type->getSubtype(static_cast<uint64_t>(column_id.value()));
+ const auto* projection = find_request_projection(*_request, column_id);
+ DORIS_CHECK(projection != nullptr);
+ std::set<uint64_t> projected_ids;
+ const bool partial = has_pruned_projection(*projection);
+ if (partial) {
+ // Match includeTypes exactly: an unread DATE sibling cannot raise a conversion
+ // error and must not disable pruning for the projected columns.
+ RETURN_IF_ERROR(collect_projected_type_ids(type, *projection, &projected_ids));
+ }
+ collect_dates(collect_dates, type, partial ? &projected_ids : nullptr);
+ }
auto builder = ::orc::SearchArgumentFactory::newBuilder();
bool has_pushdown = false;
builder->startAnd();
@@ -1670,8 +1722,45 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
}
std::vector<int> sarg_needed_stripes;
+ std::set<uint64_t> unsafe_date_stripes;
try {
+ if (!_state->decoded_date_column_ids.empty()) {
+ // getStripeStatistics also reads every ROW_INDEX stream. A metadata-only SARG
+ // proves which stripes cannot contain an invalid DATE without touching skipped data
+ // or exposing malformed, unprojected row-index streams to the SDK.
+ constexpr auto date_type = ::orc::PredicateDataType::DATE;
+ auto builder = ::orc::SearchArgumentFactory::newBuilder();
+ builder->startOr();
+ for (const auto column_id : _state->decoded_date_column_ids) {
+ builder->lessThan(column_id, date_type, ::orc::Literal(date_type, EPOCH_DAYS_MIN));
+ builder->startNot();
+ builder->lessThanEquals(column_id, date_type,
+ ::orc::Literal(date_type, EPOCH_DAYS_MAX));
+ builder->end();
+ builder->equals(column_id, date_type,
+ ::orc::Literal(date_type, EPOCH_DAYS_0000_02_29));
+ }
+ builder->end();
+ // The SDK non-const copy constructor transfers ownership instead of copying.
+ ::orc::RowReaderOptions validation_options(std::as_const(_state->row_reader_options));
+ validation_options.searchArgument(builder->build());
+ const auto needs_validation = _state->reader->getNeedReadStripes(validation_options);
+ // The SDK caches a borrowed evaluator for its next row reader. Consume it while
+ // validation_options is alive, before installing the query's actual SARG.
+ _state->reader->createRowReader(validation_options).reset();
+ for (const auto stripe_index : split_stripes) {
+ if (stripe_index >= needs_validation.size() ||
+ needs_validation[stripe_index] != 0) {
+ unsafe_date_stripes.insert(stripe_index);
+ }
+ }
+ }
sarg_needed_stripes = _state->reader->getNeedReadStripes(_state->row_reader_options);
+ if (!unsafe_date_stripes.empty()) {
+ // getNeedReadStripes caches its SARG evaluator for the first row reader, even when
+ // that reader has no SARG. Consume the cache before opening an unsafe range.
+ _state->reader->createRowReader(_state->row_reader_options).reset();
+ }
} catch (const Exception& e) {
if (is_orc_stop(_io_ctx.get(), e)) {
return Status::EndOfFile("stop");
@@ -1695,7 +1784,8 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
int64_t filtered_bytes = 0;
for (const auto stripe_index : split_stripes) {
bool drop = false;
- if (stripe_index < sarg_needed_stripes.size() && sarg_needed_stripes[stripe_index] == 0) {
+ if (!unsafe_date_stripes.contains(stripe_index) &&
+ stripe_index < sarg_needed_stripes.size() && sarg_needed_stripes[stripe_index] == 0) {
drop = true;
}
if (!drop) {
@@ -1712,7 +1802,7 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
}
}
- if (filtered_stripes == 0) {
+ if (filtered_stripes == 0 && unsafe_date_stripes.empty()) {
return Status::OK();
}
@@ -1739,6 +1829,7 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
.last_stripe = last_stripe,
.offset = offset,
.length = end_offset - offset,
+ .disable_sarg = unsafe_date_stripes.contains(first_stripe),
});
} catch (const std::exception& e) {
return Status::InternalError("Failed to build ORC stripe read range: {}", e.what());
@@ -1750,7 +1841,8 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
uint64_t previous = range_first;
for (size_t idx = 1; idx < selected_stripes.size(); ++idx) {
const auto stripe_index = selected_stripes[idx];
- if (stripe_index == previous + 1) {
+ if (stripe_index == previous + 1 &&
+ unsafe_date_stripes.contains(stripe_index) == unsafe_date_stripes.contains(previous)) {
previous = stripe_index;
continue;
}
@@ -1790,9 +1882,18 @@ Status OrcReader::_create_row_reader() {
if (_state->orc_lazy_read_enabled && _orc_filter == nullptr) {
_orc_filter = std::make_unique<OrcFilterImpl>(this);
}
+ // The SDK retains its SARG by reference, so the active options must outlive the reader.
+ _state->row_reader.reset();
+ auto& options = _state->active_row_reader_options;
+ options = _state->row_reader_options;
+ if (_state->stripe_pruning_applied &&
+ _state->selected_stripe_ranges[_state->current_stripe_range].disable_sarg) {
+ // The SDK also prunes row groups; clearing only a copy lets later safe ranges keep
+ // their SARG while the retained unsafe range reaches DATE decoding.
+ options.searchArgument(nullptr);
+ }
_state->row_reader = _state->reader->createRowReader(
- _state->row_reader_options,
- _state->orc_lazy_read_enabled ? _orc_filter.get() : nullptr);
+ options, _state->orc_lazy_read_enabled ? _orc_filter.get() : nullptr);
_state->selected_type = &_state->row_reader->getSelectedType();
DORIS_CHECK(_state->selected_type->getKind() == ::orc::TypeKind::STRUCT);
// Row-id fetch seeks before every read; a one-row batch preserves exact selection instead
@@ -2275,6 +2376,17 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque
_state->root_type->getSubtype(static_cast<uint64_t>(count_projection.local_id()));
DORIS_CHECK(count_type != nullptr);
+ std::vector<uint32_t> date_column_ids;
+ const auto collect_dates = [&](auto&& self, const ::orc::Type& type) -> void {
+ if (type.getKind() == ::orc::TypeKind::DATE) {
+ date_column_ids.push_back(cast_set<uint32_t>(type.getColumnId()));
+ }
+ for (uint64_t i = 0; i < type.getSubtypeCount(); ++i) {
+ self(self, *type.getSubtype(i));
+ }
+ };
+ collect_dates(collect_dates, *count_type);
+
result->count = 0;
const auto stripe_statistics_count = _state->reader->getNumberOfStripeStatistics();
for (const auto stripe_index : selected_stripes) {
@@ -2294,6 +2406,26 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque
return Status::NotSupported("Missing ORC stripe statistics for stripe {}",
stripe_index);
}
+ // COUNT skips decoding even for complex arguments. Every DATE leaf must be safe,
+ // including an unrepresentable day inside otherwise valid bounds, to preserve errors.
+ for (const auto column_id : date_column_ids) {
+ // Schema IDs can outlive truncated stripe statistics; the SDK lookup is unchecked.
+ if (column_id >= stripe_statistics->getNumberOfColumns()) {
+ return Status::NotSupported("Missing ORC DATE statistics for COUNT");
+ }
+ const auto* date_statistics = dynamic_cast<const ::orc::DateColumnStatistics*>(
+ stripe_statistics->getColumnStatistics(column_id));
+ if (date_statistics == nullptr ||
+ (date_statistics->getNumberOfValues() != 0 &&
+ (!date_statistics->hasMinimum() || !date_statistics->hasMaximum() ||
+ !epoch_days_range_is_representable(date_statistics->getMinimum(),
+ date_statistics->getMaximum())))) {
+ return Status::NotSupported("ORC DATE COUNT requires value validation");
+ }
+ }
+ if (count_type->getColumnId() >= stripe_statistics->getNumberOfColumns()) {
+ return Status::NotSupported("Missing ORC COUNT column statistics");
+ }
const auto* column_statistics = stripe_statistics->getColumnStatistics(
cast_set<uint32_t>(count_type->getColumnId()));
if (column_statistics == nullptr) {
@@ -2353,6 +2485,9 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque
return Status::NotSupported("Missing ORC stripe statistics for stripe {}",
stripe_index);
}
+ if (leaf_type->getColumnId() >= stripe_statistics->getNumberOfColumns()) {
+ return Status::NotSupported("Missing ORC min/max column statistics");
+ }
const auto* column_statistics = stripe_statistics->getColumnStatistics(
cast_set<uint32_t>(leaf_type->getColumnId()));
if (column_statistics == nullptr) {
@@ -2409,8 +2544,15 @@ Status OrcReader::_decode_column_into_block(const ::orc::StructVectorBatch& stru
const auto* selected_type = _state->selected_type->getSubtype(selected_batch_idx);
DORIS_CHECK(selected_type != nullptr);
auto column = file_block->get_by_position(block_position.value()).column->assert_mutable();
- RETURN_IF_ERROR(_decode_column(*type, *selected_type, *struct_batch.fields[selected_batch_idx],
- column, rows, selected_rows));
+ auto status = _decode_column(*type, *selected_type, *struct_batch.fields[selected_batch_idx],
+ column, rows, selected_rows);
+ if (!status.ok()) {
+ // The decoders work on a bare value buffer and cannot name the column themselves; without
+ // this the user only learns that some DATE/TIMESTAMP in some file is unrepresentable.
+ return status.prepend(fmt::format(
+ "Failed to decode ORC column '{}': ",
+ _state->root_type->getFieldName(static_cast<uint64_t>(file_column_id.value()))));
+ }
file_block->replace_by_position(block_position.value(), std::move(column));
return Status::OK();
}
diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp
index ff493bc071f..917b17083a8 100644
--- a/be/src/format_v2/parquet/parquet_reader.cpp
+++ b/be/src/format_v2/parquet/parquet_reader.cpp
@@ -1175,6 +1175,49 @@ Status ParquetReader::get_aggregate_result(const format::FileAggregateRequest& r
}
const auto& count_projection = request.columns[0].projection;
const auto& root_schema = projected_root_schema(_state->file_schema, count_projection);
+ std::vector<const ParquetColumnSchema*> date_leaves;
+ if (_state->enable_strict_mode ||
+ remove_nullable(root_schema.type)->get_primitive_type() == TYPE_DATEV2) {
+ const auto collect_dates = [&](auto&& self, const ParquetColumnSchema& schema,
+ const format::LocalColumnIndex* projection) -> void {
+ if (schema.kind == ParquetColumnSchemaKind::PRIMITIVE) {
+ if (remove_nullable(schema.type)->get_primitive_type() == TYPE_DATEV2) {
+ date_leaves.push_back(&schema);
+ }
+ return;
+ }
+ for (const auto& child : schema.children) {
+ if (format::is_child_projected(projection, child->local_id)) {
+ self(self, *child,
+ format::find_child_projection(projection, child->local_id));
+ }
+ }
+ };
+ collect_dates(collect_dates, root_schema, &count_projection);
+ }
+ for (const auto* date_leaf : date_leaves) {
+ // Definition levels count physical NULLs only. DATE conversion can add logical NULLs
+ // or raise an error, including for nested children in strict mode. Validate every
+ // projected DATE leaf before allowing a levels-only count to bypass conversion.
+ for (const auto& row_group_plan : _state->scan_plan->row_groups) {
+ const auto& group = _state->file_context.native_metadata->to_thrift()
+ .row_groups[row_group_plan.row_group_id];
+ const auto& chunk = group.columns[date_leaf->leaf_column_id];
+ if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics) {
+ return Status::NotSupported("Parquet DATE COUNT requires value validation");
+ }
+ const auto stats = detail::sanitize_native_footer_statistics(
+ date_leaf->type_descriptor, chunk.meta_data.statistics,
+ detail::has_supported_type_defined_order(
+ _state->file_context.native_metadata->to_thrift(),
+ date_leaf->leaf_column_id));
+ const auto decoded = ParquetStatisticsUtils::TransformColumnStatistics(
+ *date_leaf, &stats, chunk.meta_data.num_values, _state->timezone);
+ if (!decoded.has_min_max && !(decoded.has_null_count && !decoded.has_not_null)) {
+ return Status::NotSupported("Parquet DATE COUNT requires value validation");
+ }
+ }
+ }
// A required primitive COUNT(col) still carries its projection so the unsupported-type
// validation above cannot be bypassed. Once validated, its definition level proves that
// every selected row is non-NULL, so preserve the already-computed footer row count and
diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp
index 6ea6fed0797..e4928e65a7c 100644
--- a/be/src/format_v2/parquet/parquet_statistics.cpp
+++ b/be/src/format_v2/parquet/parquet_statistics.cpp
@@ -1661,6 +1661,16 @@ bool set_native_page_scalar_min_max(const tparquet::ColumnIndex& column_index,
}
const auto min_value = unaligned_load<ValueType>(column_index.min_values[page_idx].data());
const auto max_value = unaligned_load<ValueType>(column_index.max_values[page_idx].data());
+ if constexpr (std::is_same_v<ValueType, int32_t>) {
+ if (remove_nullable(column_schema.type)->get_primitive_type() == TYPE_DATEV2 &&
+ !epoch_days_range_is_representable(min_value, max_value)) {
+ // An invalid interior DATE errors in strict mode and becomes NULL otherwise.
+ // Neither min/max nor physical null counts prove the materialized result here.
+ // Retain this page without discarding usable statistics on the other pages.
+ *page_statistics = {};
+ return true;
+ }
+ }
if constexpr (std::is_integral_v<ValueType>) {
if (remove_nullable(column_schema.type)->get_primitive_type() == TYPE_TIMEV2) {
int64_t units_per_day = 0;
diff --git a/be/test/core/data_type_serde/data_type_datev2_serde_calendar_test.cpp b/be/test/core/data_type_serde/data_type_datev2_serde_calendar_test.cpp
new file mode 100644
index 00000000000..a03784222bc
--- /dev/null
+++ b/be/test/core/data_type_serde/data_type_datev2_serde_calendar_test.cpp
@@ -0,0 +1,358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Doris DATE values are numbered in MySQL's calendar, where year 0 is not a leap year. Arrow
+// `date32`, Parquet `DATE` and ORC `DATE` are all days since 1970-01-01 in the proleptic
+// Gregorian calendar, where year 0 IS a leap year. These tests pin the boundary so the two
+// numberings cannot drift apart again: 0000-01-01 must leave Doris as -719528, not -719527.
+
+#include <arrow/api.h>
+#include <cctz/time_zone.h>
+#include <gtest/gtest.h>
+
+#include <limits>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/assert_cast.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type_serde/data_type_datev2_serde.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/value/vdatetime_value.h"
+
+namespace doris {
+
+namespace {
+
+struct DateCase {
+ int year;
+ int month;
+ int day;
+ int32_t epoch_days;
+};
+
+// 0000-01-01 .. 0000-02-28 are the only dates where Doris's daynr and the proleptic Gregorian
+// ordinal disagree, so the set brackets that window on both sides.
+const std::vector<DateCase>& boundary_cases() {
+ static const std::vector<DateCase> cases = {
+ {0, 1, 1, -719528}, {0, 2, 28, -719470}, {0, 3, 1, -719468}, {1, 1, 1, -719162},
+ {1969, 12, 31, -1}, {1970, 1, 1, 0}, {2024, 1, 1, 19723}, {9999, 12, 31, 2932896}};
+ return cases;
+}
+
+DateV2Value<DateV2ValueType> make_date(int year, int month, int day) {
+ DateV2Value<DateV2ValueType> value;
+ value.unchecked_set_time(year, month, day, 0, 0, 0, 0);
+ return value;
+}
+
+ColumnDateV2::MutablePtr boundary_column() {
+ auto column = ColumnDateV2::create();
+ auto& data = column->get_data();
+ for (const auto& c : boundary_cases()) {
+ auto value = make_date(c.year, c.month, c.day);
+ data.push_back(*reinterpret_cast<UInt32*>(&value));
+ }
+ return column;
+}
+
+std::shared_ptr<arrow::Array> build_date32(const std::vector<int32_t>& days) {
+ arrow::Date32Builder builder;
+ for (int32_t d : days) {
+ EXPECT_TRUE(builder.Append(d).ok());
+ }
+ std::shared_ptr<arrow::Array> array;
+ EXPECT_TRUE(builder.Finish(&array).ok());
+ return array;
+}
+
+} // namespace
+
+class DataTypeDateV2SerDeCalendarTest : public ::testing::Test {
+protected:
+ DataTypeDateV2SerDe serde;
+ cctz::time_zone tz = cctz::utc_time_zone();
+};
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, WriteArrowUsesProlepticGregorian) {
+ auto column = boundary_column();
+ arrow::Date32Builder builder;
+ ASSERT_TRUE(serde.write_column_to_arrow(*column, nullptr, &builder, 0,
+ static_cast<int64_t>(column->size()), tz)
+ .ok());
+ std::shared_ptr<arrow::Array> array;
+ ASSERT_TRUE(builder.Finish(&array).ok());
+
+ const auto& date32 = assert_cast<const arrow::Date32Array&>(*array);
+ ASSERT_EQ(boundary_cases().size(), static_cast<size_t>(date32.length()));
+ for (size_t i = 0; i < boundary_cases().size(); ++i) {
+ EXPECT_EQ(boundary_cases()[i].epoch_days, date32.Value(static_cast<int64_t>(i)))
+ << "row " << i;
+ }
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, WriteArrowHonoursNullMap) {
+ auto column = boundary_column();
+ NullMap null_map;
+ null_map.resize_fill(column->size(), 0);
+ null_map[0] = 1; // 0000-01-01
+ null_map[2] = 1; // 0000-03-01
+
+ arrow::Date32Builder builder;
+ ASSERT_TRUE(serde.write_column_to_arrow(*column, &null_map, &builder, 0,
+ static_cast<int64_t>(column->size()), tz)
+ .ok());
+ std::shared_ptr<arrow::Array> array;
+ ASSERT_TRUE(builder.Finish(&array).ok());
+
+ const auto& date32 = assert_cast<const arrow::Date32Array&>(*array);
+ for (size_t i = 0; i < boundary_cases().size(); ++i) {
+ if (null_map[i]) {
+ EXPECT_TRUE(date32.IsNull(static_cast<int64_t>(i))) << "row " << i;
+ } else {
+ EXPECT_EQ(boundary_cases()[i].epoch_days, date32.Value(static_cast<int64_t>(i)))
+ << "row " << i;
+ }
+ }
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, ReadDate32RestoresTheSameCalendarDay) {
+ std::vector<int32_t> days;
+ for (const auto& c : boundary_cases()) {
+ days.push_back(c.epoch_days);
+ }
+ auto array = build_date32(days);
+
+ auto column = ColumnDateV2::create();
+ ASSERT_TRUE(serde.read_column_from_arrow(*column, array.get(), 0, array->length(), tz).ok());
+
+ ASSERT_EQ(boundary_cases().size(), column->size());
+ const auto& values = column->get_data();
+ for (size_t i = 0; i < boundary_cases().size(); ++i) {
+ const auto& c = boundary_cases()[i];
+ EXPECT_EQ(c.year, values[i].year()) << "row " << i;
+ EXPECT_EQ(c.month, values[i].month()) << "row " << i;
+ EXPECT_EQ(c.day, values[i].day()) << "row " << i;
+ }
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, ArrowRoundTripIsLossless) {
+ auto column = boundary_column();
+ arrow::Date32Builder builder;
+ ASSERT_TRUE(serde.write_column_to_arrow(*column, nullptr, &builder, 0,
+ static_cast<int64_t>(column->size()), tz)
+ .ok());
+ std::shared_ptr<arrow::Array> array;
+ ASSERT_TRUE(builder.Finish(&array).ok());
+
+ auto restored = ColumnDateV2::create();
+ ASSERT_TRUE(serde.read_column_from_arrow(*restored, array.get(), 0, array->length(), tz).ok());
+ ASSERT_EQ(column->size(), restored->size());
+ for (size_t i = 0; i < column->size(); ++i) {
+ EXPECT_EQ(column->get_data()[i], restored->get_data()[i]) << "row " << i;
+ }
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, ReadDate32RejectsUnrepresentableDays) {
+ const auto expect_rejected = [&](int32_t days, const char* why) {
+ auto array = build_date32({days});
+ auto column = ColumnDateV2::create();
+ const auto status = serde.read_column_from_arrow(*column, array.get(), 0, 1, tz);
+ EXPECT_FALSE(status.ok()) << why << " (days=" << days << ")";
+ EXPECT_NE(std::string::npos, status.to_string().find("outside the Doris DATE range"))
+ << status.to_string();
+ };
+
+ expect_rejected(-719529, "one day before 0000-01-01");
+ // 0000-02-29 exists in the proleptic Gregorian calendar but Doris has no such date. The old
+ // `encoded + 719528` mapping decoded it as 0000-02-28; it must be rejected instead.
+ expect_rejected(-719469, "proleptic-only leap day 0000-02-29");
+ expect_rejected(2932897, "one day after 9999-12-31");
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, ReadDate64UsesTheSameCalendar) {
+ constexpr int64_t millis_per_day = 24LL * 60 * 60 * 1000;
+ arrow::Date64Builder builder;
+ for (const auto& c : boundary_cases()) {
+ ASSERT_TRUE(builder.Append(static_cast<int64_t>(c.epoch_days) * millis_per_day).ok());
+ }
+ std::shared_ptr<arrow::Array> array;
+ ASSERT_TRUE(builder.Finish(&array).ok());
+
+ auto column = ColumnDateV2::create();
+ ASSERT_TRUE(serde.read_column_from_arrow(*column, array.get(), 0, array->length(), tz).ok());
+
+ ASSERT_EQ(boundary_cases().size(), column->size());
+ const auto& values = column->get_data();
+ for (size_t i = 0; i < boundary_cases().size(); ++i) {
+ const auto& c = boundary_cases()[i];
+ EXPECT_EQ(c.year, values[i].year()) << "row " << i;
+ EXPECT_EQ(c.month, values[i].month()) << "row " << i;
+ EXPECT_EQ(c.day, values[i].day()) << "row " << i;
+ }
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, NestedArrayOfDateUsesTheSameCalendar) {
+ // ARRAY<DATE> has no encoding of its own; it delegates every element to the DATE SerDe. The
+ // reported bug showed up through this path too, so keep it covered.
+ // DataTypeArray stores its element type as DataTypeNullablePtr, so the nested column has to be
+ // a ColumnNullable for the element SerDe to match.
+ auto nested = boundary_column();
+ const auto element_count = nested->size();
+ auto element_null_map = ColumnUInt8::create();
+ element_null_map->get_data().resize_fill(element_count, 0);
+ auto nullable_nested = ColumnNullable::create(std::move(nested), std::move(element_null_map));
+ auto offsets = ColumnArray::ColumnOffsets::create();
+ offsets->get_data().push_back(static_cast<ColumnArray::Offset64>(element_count));
+ auto array_column = ColumnArray::create(std::move(nullable_nested), std::move(offsets));
+
+ auto array_type = std::make_shared<DataTypeArray>(std::make_shared<DataTypeDateV2>());
+ auto array_serde = array_type->get_serde();
+
+ auto value_builder = std::make_shared<arrow::Date32Builder>();
+ arrow::ListBuilder list_builder(arrow::default_memory_pool(), value_builder);
+ ASSERT_TRUE(array_serde->write_column_to_arrow(*array_column, nullptr, &list_builder, 0, 1, tz)
+ .ok());
+ std::shared_ptr<arrow::Array> arrow_array;
+ ASSERT_TRUE(list_builder.Finish(&arrow_array).ok());
+
+ const auto& list = assert_cast<const arrow::ListArray&>(*arrow_array);
+ const auto& values = assert_cast<const arrow::Date32Array&>(*list.values());
+ ASSERT_EQ(boundary_cases().size(), static_cast<size_t>(values.length()));
+ for (size_t i = 0; i < boundary_cases().size(); ++i) {
+ EXPECT_EQ(boundary_cases()[i].epoch_days, values.Value(static_cast<int64_t>(i)))
+ << "element " << i;
+ }
+}
+
+// The Parquet and ORC readers do not go through Arrow: rows, dictionary entries and column
+// statistics all land in `read_column_from_decoded_values()`. Cover that entry point with the same
+// boundary set, in both the strict and the null-on-failure mode the file scanners use.
+TEST_F(DataTypeDateV2SerDeCalendarTest, ReadDecodedValuesUsesTheSameCalendar) {
+ std::vector<int32_t> values;
+ for (const auto& c : boundary_cases()) {
+ values.push_back(c.epoch_days);
+ }
+ values.push_back(0); // payload of the null row below, never decoded
+ std::vector<uint8_t> null_map(values.size(), 0);
+ null_map.back() = 1;
+
+ DecodedColumnView view;
+ view.value_kind = DecodedValueKind::INT32;
+ view.row_count = static_cast<int64_t>(values.size());
+ view.values = reinterpret_cast<const uint8_t*>(values.data());
+ view.null_map = null_map.data();
+
+ auto column = ColumnDateV2::create();
+ ASSERT_TRUE(serde.read_column_from_decoded_values(*column, view).ok());
+ ASSERT_EQ(values.size(), column->size());
+ const auto& data = column->get_data();
+ for (size_t i = 0; i < boundary_cases().size(); ++i) {
+ const auto& c = boundary_cases()[i];
+ EXPECT_EQ(c.year, data[i].year()) << "row " << i;
+ EXPECT_EQ(c.month, data[i].month()) << "row " << i;
+ EXPECT_EQ(c.day, data[i].day()) << "row " << i;
+ }
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, ReadDecodedValuesRejectsUnrepresentableDaysWhenStrict) {
+ // -719469 is the proleptic-only 0000-02-29: the only value inside the file-format range that
+ // Doris cannot represent, and the one the old dictionary fallback decoded as 0000-02-28.
+ const std::vector<int32_t> values = {-719528, -719470, -719469, -719468, 19723};
+ DecodedColumnView view;
+ view.value_kind = DecodedValueKind::INT32;
+ view.row_count = static_cast<int64_t>(values.size());
+ view.values = reinterpret_cast<const uint8_t*>(values.data());
+ view.enable_strict_mode = true;
+
+ auto column = ColumnDateV2::create();
+ const auto status = serde.read_column_from_decoded_values(*column, view);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(std::string::npos, status.to_string().find("outside the Doris DATE range"))
+ << status.to_string();
+ EXPECT_NE(std::string::npos, status.to_string().find("-719469")) << status.to_string();
+ // A failed batch must not leave a half-written column behind.
+ EXPECT_EQ(0, column->size());
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, ReadDecodedValuesNullsOnlyTheUnrepresentableRow) {
+ const std::vector<int32_t> values = {-719528, -719470, -719469, -719468, 19723};
+ NullMap conversion_failures;
+ conversion_failures.resize_fill(values.size(), 0);
+
+ DecodedColumnView view;
+ view.value_kind = DecodedValueKind::INT32;
+ view.row_count = static_cast<int64_t>(values.size());
+ view.values = reinterpret_cast<const uint8_t*>(values.data());
+ view.conversion_failure_null_map = &conversion_failures;
+
+ auto column = ColumnDateV2::create();
+ ASSERT_TRUE(serde.read_column_from_decoded_values(*column, view).ok());
+ ASSERT_EQ(values.size(), column->size());
+ const std::vector<uint8_t> expected_failures = {0, 0, 1, 0, 0};
+ for (size_t i = 0; i < expected_failures.size(); ++i) {
+ EXPECT_EQ(expected_failures[i], conversion_failures[i]) << "row " << i;
+ }
+ const auto& data = column->get_data();
+ EXPECT_EQ(0, data[0].year());
+ EXPECT_EQ(1, data[0].month());
+ EXPECT_EQ(1, data[0].day());
+ EXPECT_EQ(0, data[1].year());
+ EXPECT_EQ(2, data[1].month());
+ EXPECT_EQ(28, data[1].day());
+ EXPECT_EQ(0, data[3].year());
+ EXPECT_EQ(3, data[3].month());
+ EXPECT_EQ(1, data[3].day());
+ EXPECT_EQ(2024, data[4].year());
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, NullDate32PayloadIsNotDecoded) {
+ auto array = build_date32({std::numeric_limits<int32_t>::min(), -719528});
+ auto data = array->data()->Copy();
+ data->buffers[0] = arrow::Buffer::FromVector(std::vector<uint8_t> {0x02});
+ data->SetNullCount(1);
+ auto nullable_array = arrow::MakeArray(data);
+ auto column = ColumnDateV2::create();
+ ASSERT_TRUE(serde.read_column_from_arrow(*column, nullable_array.get(), 0, 2, tz).ok());
+ ASSERT_EQ(column->size(), 2);
+ EXPECT_EQ(column->get_data()[1], make_date(0, 1, 1));
+}
+
+TEST_F(DataTypeDateV2SerDeCalendarTest, EveryRepresentableDateMatchesCivilCalendar) {
+ const cctz::civil_day epoch(1970, 1, 1);
+ for (auto civil = cctz::civil_day(0, 1, 1); civil < cctz::civil_day(10000, 1, 1); ++civil) {
+ // The external calendar has this leap day, while Doris intentionally follows MySQL.
+ if (civil.year() == 0 && civil.month() == 2 && civil.day() == 29) {
+ continue;
+ }
+ const auto value = make_date(civil.year(), civil.month(), civil.day());
+ const auto days = civil - epoch;
+ ASSERT_EQ(days, daynr_to_epoch_days(value.daynr())) << value.to_string();
+ ASSERT_EQ(value.daynr(), epoch_days_to_daynr(days)) << value.to_string();
+ }
+ EXPECT_EQ(epoch_days_to_daynr(-719469), 0);
+ EXPECT_EQ(epoch_days_to_daynr(std::numeric_limits<int64_t>::min()), 0);
+ EXPECT_EQ(epoch_days_to_daynr(std::numeric_limits<int64_t>::max()), 0);
+}
+
+} // namespace doris
diff --git a/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp b/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp
index 8595b19e029..f6909877350 100644
--- a/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp
+++ b/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp
@@ -20,6 +20,7 @@
#include <gtest/gtest.h>
#include <limits>
+#include <tuple>
#include "core/data_type/data_type_date_or_datetime_v2.h"
#include "core/data_type/data_type_timestamptz.h"
@@ -721,4 +722,30 @@ TEST_F(PartitionTransformersTest, test_nullable_column_string_truncate_transform
EXPECT_EQ("db", result_strings->get_data_at(2).to_string());
}
+TEST_F(PartitionTransformersTest, DatePartitionsMatchYearZeroFileOrdinals) {
+ auto column = ColumnDateV2::create();
+ const std::vector<std::tuple<int, int, int, int64_t>> cases = {
+ {0, 1, 1, -719528}, {0, 2, 28, -719470}, {0, 3, 1, -719468},
+ {1969, 12, 31, -1}, {1970, 1, 1, 0}, {2024, 1, 1, 19723}};
+ for (const auto& [year, month, day, ordinal] : cases) {
+ DateV2Value<DateV2ValueType> value;
+ value.unchecked_set_time(year, month, day, 0, 0, 0, 0);
+ column->get_data().push_back(value);
+ }
+ auto type = std::make_shared<DataTypeDateV2>();
+ Block block({{std::move(column), type, "d"}});
+ DateDayPartitionColumnTransform day_transform(type);
+ DateBucketPartitionColumnTransform bucket_transform(type, 16);
+ const auto days = day_transform.apply(block, 0);
+ const auto buckets = bucket_transform.apply(block, 0);
+ for (size_t i = 0; i < cases.size(); ++i) {
+ // Iceberg hashes a signed, eight-byte epoch day, including for DATE's INT32 carrier.
+ const int64_t ordinal = std::get<3>(cases[i]);
+ const auto hash = HashUtil::murmur_hash3_32(&ordinal, sizeof(ordinal), 0);
+ EXPECT_EQ(ordinal, assert_cast<const ColumnInt32&>(*days.column).get_data()[i]);
+ EXPECT_EQ((hash & INT32_MAX) % 16,
+ assert_cast<const ColumnInt32&>(*buckets.column).get_data()[i]);
+ }
+}
+
} // namespace doris
diff --git a/be/test/exec/test_data/orc_scanner/date_count_truncated_stats.orc b/be/test/exec/test_data/orc_scanner/date_count_truncated_stats.orc
new file mode 100644
index 00000000000..7c942f141f3
Binary files /dev/null and b/be/test/exec/test_data/orc_scanner/date_count_truncated_stats.orc differ
diff --git a/be/test/exec/test_data/orc_scanner/date_large_ordinal.orc b/be/test/exec/test_data/orc_scanner/date_large_ordinal.orc
new file mode 100644
index 00000000000..b0c21072394
Binary files /dev/null and b/be/test/exec/test_data/orc_scanner/date_large_ordinal.orc differ
diff --git a/be/test/exec/test_data/orc_scanner/date_metadata_fixtures.md b/be/test/exec/test_data/orc_scanner/date_metadata_fixtures.md
new file mode 100644
index 00000000000..47bbff013f7
--- /dev/null
+++ b/be/test/exec/test_data/orc_scanner/date_metadata_fixtures.md
@@ -0,0 +1,39 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# ORC DATE metadata fixtures
+
+These small, uncompressed files exercise metadata validation independently of
+DATE value decoding. They were generated with PyArrow 24.0.0's ORC writer.
+
+- `date_preflight.orc`: schema `struct<id:int,d:date,unused:string>`, two rows
+ `(0, DATE ordinal 0, 'a')` and `(0, DATE ordinal 1, 'b')`, one stripe, and
+ row-index stride 1000. A predicate `id > 7` excludes the stripe. Projecting
+ `d` must not add stripe-footer or row-index reads.
+- `date_preflight_bad_index.orc`: identical bytes except that the unused
+ column's `ROW_INDEX` stream column ID in the stripe footer is changed from
+ 3 to 100. The stream itself remains nonempty and valid. No data or index in
+ this excluded stripe should be read or parsed during DATE preflight.
+- `date_count_truncated_stats.orc`: schema `struct<id:int,s:struct<d:date>>`,
+ two rows with IDs 0 and 1 and DATE ordinals 0 and 1. The stripe metadata's
+ fourth `colStats` entry (nested DATE ID 3) is removed, while the schema and
+ file-level statistics remain complete. Row-index streams are removed, the
+ stripe index length and file row-index stride are zero, and footer lengths
+ are adjusted. `COUNT(s)` must decline metadata pushdown without an unchecked
+ lookup; ordinary reading still returns both intact struct values.
diff --git a/be/test/exec/test_data/orc_scanner/date_out_of_range.orc b/be/test/exec/test_data/orc_scanner/date_out_of_range.orc
new file mode 100644
index 00000000000..9a7acdd055b
Binary files /dev/null and b/be/test/exec/test_data/orc_scanner/date_out_of_range.orc differ
diff --git a/be/test/exec/test_data/orc_scanner/date_preflight.orc b/be/test/exec/test_data/orc_scanner/date_preflight.orc
new file mode 100644
index 00000000000..41350b1c924
Binary files /dev/null and b/be/test/exec/test_data/orc_scanner/date_preflight.orc differ
diff --git a/be/test/exec/test_data/orc_scanner/date_preflight_bad_index.orc b/be/test/exec/test_data/orc_scanner/date_preflight_bad_index.orc
new file mode 100644
index 00000000000..3b8e2e5f7be
Binary files /dev/null and b/be/test/exec/test_data/orc_scanner/date_preflight_bad_index.orc differ
diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp
index e3e6d686cea..1817793936e 100644
--- a/be/test/format_v2/orc/orc_reader_test.cpp
+++ b/be/test/format_v2/orc/orc_reader_test.cpp
@@ -150,8 +150,9 @@ DateV2Value<DateV2ValueType> make_date_v2(uint16_t year, uint8_t month, uint8_t
}
int64_t orc_date_offset(uint16_t year, uint8_t month, uint8_t day) {
- static constexpr int32_t DATE_THRESHOLD = 719528;
- return make_date_v2(year, month, day).daynr() - DATE_THRESHOLD;
+ // ORC DATE is days since 1970-01-01 in the proleptic Gregorian calendar, which is not the
+ // same as Doris's daynr minus the epoch daynr for year-zero dates.
+ return daynr_to_epoch_days(make_date_v2(year, month, day).daynr());
}
DateV2Value<DateTimeV2ValueType> make_datetime_v2(uint16_t year, uint8_t month, uint8_t day,
@@ -3339,7 +3340,6 @@ void write_complex_orc_file(const std::string& file_path) {
void write_map_decimal_date_orc_file(const std::string& file_path) {
constexpr size_t ROWS = 4;
- constexpr int64_t HIVE_012_1900_DAY_OFFSET = -719530;
constexpr int64_t YEAR_0000_12_29_DAY_OFFSET = -719165;
constexpr int64_t YEAR_1000_10_16_DAY_OFFSET = -353997;
@@ -3371,7 +3371,9 @@ void write_map_decimal_date_orc_file(const std::string& file_path) {
key_batch.values[1] = 9999999999L;
key_batch.values[2] = 0;
key_batch.values[3] = 1;
- value_batch.data[0] = HIVE_012_1900_DAY_OFFSET;
+ // The smallest DATE Doris can represent: -719528 in the proleptic Gregorian calendar the ORC
+ // spec defines DATE in, one day below what Doris's own MySQL-calendar daynr would suggest.
+ value_batch.data[0] = orc_date_offset(0, 1, 1);
value_batch.data[1] = orc_date_offset(9999, 12, 31);
value_batch.data[2] = YEAR_0000_12_29_DAY_OFFSET;
value_batch.data[3] = YEAR_1000_10_16_DAY_OFFSET;
@@ -3389,6 +3391,42 @@ void write_map_decimal_date_orc_file(const std::string& file_path) {
out.write(memory_stream.getData(), static_cast<std::streamsize>(memory_stream.getLength()));
}
+// A flat `struct<id:int,d:date>` file whose DATE ordinals are written verbatim, so a test can
+// place a value that no Doris DATE maps to (the proleptic-only 0000-02-29, or an ordinal outside
+// the type's range) next to representable ones.
+void write_date_orc_file(const std::string& file_path,
+ const std::vector<std::optional<int64_t>>& day_offsets) {
+ auto type =
+ std::unique_ptr<::orc::Type>(::orc::Type::buildTypeFromString("struct<id:int,d:date>"));
+
+ MemoryOutputStream memory_stream(1024 * 1024);
+ ::orc::WriterOptions options;
+ options.setCompression(::orc::CompressionKind_NONE);
+ options.setMemoryPool(::orc::getDefaultPool());
+ auto writer = ::orc::createWriter(*type, &memory_stream, options);
+ auto batch = writer->createRowBatch(day_offsets.size());
+ auto& struct_batch = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto& id_batch = dynamic_cast<::orc::LongVectorBatch&>(*struct_batch.fields[0]);
+ auto& date_batch = dynamic_cast<::orc::LongVectorBatch&>(*struct_batch.fields[1]);
+
+ date_batch.hasNulls = true;
+ for (size_t row = 0; row < day_offsets.size(); ++row) {
+ id_batch.data[row] = static_cast<int64_t>(row) + 1;
+ date_batch.notNull[row] = day_offsets[row].has_value() ? 1 : 0;
+ // Deliberately garbage under a null slot: a decoder must never look at it.
+ date_batch.data[row] = day_offsets[row].value_or(std::numeric_limits<int64_t>::min());
+ }
+
+ struct_batch.numElements = day_offsets.size();
+ id_batch.numElements = day_offsets.size();
+ date_batch.numElements = day_offsets.size();
+ writer->add(*batch);
+ writer->close();
+
+ std::ofstream out(file_path, std::ios::binary);
+ out.write(memory_stream.getData(), static_cast<std::streamsize>(memory_stream.getLength()));
+}
+
void write_two_stripe_orc_array_map_file(const std::string& file_path) {
auto type = std::unique_ptr<::orc::Type>(::orc::Type::buildTypeFromString(
"struct<array_col:array<int>,map_col:map<string,int>,payload:string>"));
@@ -11096,12 +11134,140 @@ TEST_F(NewOrcReaderTest, ReadMapDecimalDateWithCenturyBoundary) {
const auto& map_column = assert_cast<const ColumnMap&>(map_nullable.get_nested_column());
ASSERT_EQ(map_column.get_offsets().size(), 4);
ASSERT_EQ(map_column.get_values().size(), 4);
- EXPECT_EQ(schema[1].children[1].type->to_string(map_column.get_values(), 0), "1900-01-01");
+ EXPECT_EQ(schema[1].children[1].type->to_string(map_column.get_values(), 0), "0000-01-01");
EXPECT_EQ(schema[1].children[1].type->to_string(map_column.get_values(), 1), "9999-12-31");
EXPECT_EQ(schema[1].children[1].type->to_string(map_column.get_values(), 2), "0000-12-29");
EXPECT_EQ(schema[1].children[1].type->to_string(map_column.get_values(), 3), "1000-10-16");
}
+// ORC DATE is a proleptic Gregorian day ordinal, so year zero is the window where Doris's own
+// MySQL-calendar day number disagrees with the file. Pin the decode there, including a null row
+// whose payload must never be looked at.
+TEST_F(NewOrcReaderTest, ReadDateYearZeroWindow) {
+ const auto file_path = (_test_dir / "date_year_zero.orc").string();
+ write_date_orc_file(file_path, {orc_date_offset(0, 1, 1), orc_date_offset(0, 2, 28),
+ orc_date_offset(0, 3, 1), std::nullopt,
+ orc_date_offset(1970, 1, 1), orc_date_offset(2024, 1, 1)});
+ auto reader = create_reader_for_path(file_path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ ASSERT_EQ(schema.size(), 2);
+
+ Block block = build_file_block(schema);
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(0), field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+
+ size_t rows = 0;
+ bool eof = false;
+ ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
+ ASSERT_EQ(rows, 6);
+
+ const auto& nullable = assert_cast<const ColumnNullable&>(*block.get_by_position(1).column);
+ const std::vector<std::string> expected = {"0000-01-01", "0000-02-28", "0000-03-01",
+ "", "1970-01-01", "2024-01-01"};
+ for (size_t row = 0; row < expected.size(); ++row) {
+ if (expected[row].empty()) {
+ EXPECT_TRUE(nullable.is_null_at(row)) << "row " << row;
+ continue;
+ }
+ ASSERT_FALSE(nullable.is_null_at(row)) << "row " << row;
+ EXPECT_EQ(expected[row], schema[1].type->to_string(nullable, row)) << "row " << row;
+ }
+}
+
+TEST_F(NewOrcReaderTest, ReadDateRejectsUnrepresentableOrdinals) {
+ // -719469 is 0000-02-29, which exists in the proleptic Gregorian calendar but not in Doris;
+ // -719530 is below 0000-01-01. Both used to decode as 1900-01-01 through the day dictionary.
+ for (const int64_t bad_offset : {int64_t {-719469}, int64_t {-719530}, int64_t {2932897}}) {
+ const auto file_path = (_test_dir / fmt::format("date_bad_{}.orc", bad_offset)).string();
+ write_date_orc_file(file_path, {orc_date_offset(2024, 1, 1), bad_offset});
+ auto reader = create_reader_for_path(file_path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ Block block = build_file_block(schema);
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(0), field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+
+ size_t rows = 0;
+ bool eof = false;
+ const auto status = reader->get_block(&block, &rows, &eof);
+ EXPECT_FALSE(status.ok()) << "offset " << bad_offset;
+ // The message must name the column and the offending value, and must not claim a format
+ // it did not come from: the helper is shared with Parquet.
+ EXPECT_NE(std::string::npos, status.to_string().find("outside the Doris DATE range"))
+ << status.to_string();
+ EXPECT_NE(std::string::npos, status.to_string().find(std::to_string(bad_offset)))
+ << status.to_string();
+ EXPECT_NE(std::string::npos, status.to_string().find("'d'")) << status.to_string();
+ EXPECT_EQ(std::string::npos, status.to_string().find("Parquet")) << status.to_string();
+ }
+}
+
+// A pushed-down MIN/MAX is answered from stripe statistics without reading a row, so the
+// statistics have to be decoded with exactly the same calendar as the rows. Decoding them through
+// the day dictionary used to report 1900-01-01 as the minimum of a file whose smallest row is
+// 0000-01-01 -- a value present in no row at all.
+TEST_F(NewOrcReaderTest, AggregatePushdownDateMinMaxAgreesWithRowDecode) {
+ const auto file_path = (_test_dir / "date_year_zero_minmax.orc").string();
+ write_date_orc_file(file_path, {orc_date_offset(0, 1, 1), orc_date_offset(0, 2, 1),
+ std::nullopt, orc_date_offset(0, 2, 28)});
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+
+ auto reader = create_reader_for_path(file_path);
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+
+ format::FileAggregateRequest aggregate_request;
+ aggregate_request.agg_type = TPushAggOp::type::MINMAX;
+ aggregate_request.columns.push_back(
+ {.projection = format::LocalColumnIndex::top_level(format::LocalColumnId(1))});
+ format::FileAggregateResult aggregate_result;
+ const auto status = reader->get_aggregate_result(aggregate_request, &aggregate_result);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(aggregate_result.columns.size(), 1);
+ EXPECT_EQ(aggregate_result.count, 4);
+ ASSERT_TRUE(aggregate_result.columns[0].has_min);
+ ASSERT_TRUE(aggregate_result.columns[0].has_max);
+ EXPECT_EQ(aggregate_result.columns[0].min_value.get<TYPE_DATEV2>(), make_date_v2(0, 1, 1));
+ EXPECT_EQ(aggregate_result.columns[0].max_value.get<TYPE_DATEV2>(), make_date_v2(0, 2, 28));
+}
+
+TEST_F(NewOrcReaderTest, AggregatePushdownDateMinMaxFallsBackForUnrepresentableBound) {
+ const auto file_path = (_test_dir / "date_minmax_fallback.orc").string();
+ // The minimum is the proleptic-only 0000-02-29: no Doris DATE bounds this file, so the
+ // statistics must be refused rather than silently rounded onto a neighbouring day.
+ write_date_orc_file(file_path, {std::optional<int64_t> {-719469}, orc_date_offset(2024, 1, 1)});
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+
+ auto reader = create_reader_for_path(file_path);
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+
+ format::FileAggregateRequest aggregate_request;
+ aggregate_request.agg_type = TPushAggOp::type::MINMAX;
+ aggregate_request.columns.push_back(
+ {.projection = format::LocalColumnIndex::top_level(format::LocalColumnId(1))});
+ format::FileAggregateResult aggregate_result;
+ const auto status = reader->get_aggregate_result(aggregate_request, &aggregate_result);
+ EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) << status;
+}
+
TEST_F(NewOrcReaderTest, ReadDeepNestedComplexTypes) {
const auto complex_file_path = (_test_dir / "deep_complex.orc").string();
write_deep_nested_complex_orc_file(complex_file_path);
@@ -11567,5 +11733,388 @@ TEST_F(NewOrcReaderTest, ReadProjectedComplexChildrenWithNulls) {
EXPECT_EQ(value_a.get_element(1), 202);
}
+TEST_F(NewOrcReaderTest, DateInteriorLeapDayDisablesAggregateAndSarg) {
+ const auto path = (_test_dir / "date_interior_leap_day.orc").string();
+ write_date_orc_file(path, {-719528, -719469, -719468});
+ for (bool pruning : {false, true}) {
+ auto reader = create_reader_for_path(path);
+ TQueryOptions options;
+ options.__set_enable_orc_filter_by_min_max(pruning);
+ RuntimeState state {options, TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->predicate_columns = {field_projection(1)};
+ request->conjuncts = {
+ VExprContext::create_shared(std::make_shared<NullableGreaterThanExpr<TYPE_DATEV2>>(
+ 0, remove_nullable(schema[1].type),
+ Field::create_field<TYPE_DATEV2>(make_date_v2(1, 1, 1)), "d"))};
+ ASSERT_TRUE(reader->open(request).ok());
+ Block block = build_file_block({schema[1]});
+ size_t rows = 0;
+ bool eof = false;
+ const auto status = reader->get_block(&block, &rows, &eof);
+ EXPECT_FALSE(status.ok()) << "pruning=" << pruning;
+ EXPECT_NE(status.to_string().find("-719469"), std::string::npos);
+ }
+ auto reader = create_reader_for_path(path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+ format::FileAggregateRequest aggregate;
+ aggregate.agg_type = TPushAggOp::type::MINMAX;
+ aggregate.columns.push_back({.projection = field_projection(1)});
+ format::FileAggregateResult result;
+ EXPECT_TRUE(reader->get_aggregate_result(aggregate, &result)
+ .is<ErrorCode::NOT_IMPLEMENTED_ERROR>());
+}
+
+TEST_F(NewOrcReaderTest, DateCountValidatesScalarAndNestedValues) {
+ const std::vector<std::vector<std::optional<int64_t>>> cases = {
+ {-719528, -719469, -719468},
+ {-719530, 0, 1},
+ {0, 1, 2932897},
+ {-719528, -719470, -719470},
+ {-719468, 0, 1},
+ {std::nullopt, std::nullopt, std::nullopt}};
+ for (size_t case_id = 0; case_id < cases.size(); ++case_id) {
+ SCOPED_TRACE(case_id);
+ const auto& days = cases[case_id];
+ const auto path = (_test_dir / "count_nested_dates.orc").string();
+ auto type = std::unique_ptr<::orc::Type>(::orc::Type::buildTypeFromString(
+ "struct<d:date,a:array<date>,s:struct<d:date>,m:map<int,date>>"));
+ MemoryOutputStream stream(1024 * 1024);
+ auto writer = ::orc::createWriter(*type, &stream, ::orc::WriterOptions());
+ auto batch = writer->createRowBatch(days.size());
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto fill = [&](auto&& self, ::orc::ColumnVectorBatch& column) -> void {
+ column.numElements = days.size();
+ if (auto* record = dynamic_cast<::orc::StructVectorBatch*>(&column)) {
+ for (auto* field : record->fields) {
+ self(self, *field);
+ }
+ } else if (auto* list = dynamic_cast<::orc::ListVectorBatch*>(&column)) {
+ for (size_t i = 0; i <= days.size(); ++i) {
+ list->offsets[i] = i;
+ }
+ self(self, *list->elements);
+ } else if (auto* map = dynamic_cast<::orc::MapVectorBatch*>(&column)) {
+ for (size_t i = 0; i <= days.size(); ++i) {
+ map->offsets[i] = i;
+ }
+ auto& keys = dynamic_cast<::orc::LongVectorBatch&>(*map->keys);
+ keys.numElements = days.size();
+ for (size_t i = 0; i < days.size(); ++i) {
+ keys.data[i] = i;
+ }
+ self(self, *map->elements);
+ } else {
+ auto& dates = dynamic_cast<::orc::LongVectorBatch&>(column);
+ dates.hasNulls = true;
+ for (size_t i = 0; i < days.size(); ++i) {
+ dates.notNull[i] = days[i].has_value();
+ dates.data[i] = days[i].value_or(0);
+ }
+ }
+ };
+ fill(fill, root);
+ writer->add(*batch);
+ writer->close();
+ std::ofstream out(path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ out.close();
+
+ for (int column_id = 0; column_id < 4; ++column_id) {
+ SCOPED_TRACE(column_id);
+ auto reader = create_reader_for_path(path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(column_id)};
+ ASSERT_TRUE(reader->open(request).ok());
+ format::FileAggregateRequest aggregate;
+ aggregate.agg_type = TPushAggOp::type::COUNT;
+ format::FileAggregateResult result;
+ ASSERT_TRUE(reader->get_aggregate_result(aggregate, &result).ok());
+ EXPECT_EQ(result.count, 3);
+ aggregate.columns.push_back({.projection = field_projection(column_id)});
+ const auto status = reader->get_aggregate_result(aggregate, &result);
+ const bool invalid = case_id < 3;
+ if (invalid) {
+ EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) << status;
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(result.count, column_id == 0 && case_id == 5 ? 0 : 3);
+ }
+ auto block = build_file_block({schema[column_id]});
+ size_t rows = 0;
+ bool eof = false;
+ const auto scan_status = reader->get_block(&block, &rows, &eof);
+ if (invalid) {
+ EXPECT_TRUE(scan_status.is<ErrorCode::DATA_QUALITY_ERROR>()) << scan_status;
+ EXPECT_NE(scan_status.to_string().find("outside the Doris DATE range"),
+ std::string::npos);
+ } else {
+ ASSERT_TRUE(scan_status.ok()) << scan_status;
+ EXPECT_EQ(rows, 3);
+ }
+ }
+ }
+}
+
+TEST_F(NewOrcReaderTest, DateSargGuardIgnoresUnprojectedStructChild) {
+ const auto path = (_test_dir / "partial_struct_date.orc").string();
+ auto type = std::unique_ptr<::orc::Type>(
+ ::orc::Type::buildTypeFromString("struct<id:int,s:struct<x:int,d:date>>"));
+ MemoryOutputStream stream(1024 * 1024);
+ auto writer = ::orc::createWriter(*type, &stream, ::orc::WriterOptions());
+ auto batch = writer->createRowBatch(3);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto& ids = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[0]);
+ auto& record = dynamic_cast<::orc::StructVectorBatch&>(*root.fields[1]);
+ auto& values = dynamic_cast<::orc::LongVectorBatch&>(*record.fields[0]);
+ auto& dates = dynamic_cast<::orc::LongVectorBatch&>(*record.fields[1]);
+ root.numElements = ids.numElements = record.numElements = values.numElements =
+ dates.numElements = 3;
+ for (int row = 0; row < 3; ++row) {
+ ids.data[row] = values.data[row] = row;
+ dates.data[row] = -719469;
+ }
+ writer->add(*batch);
+ writer->close();
+ std::ofstream out(path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ out.close();
+
+ for (bool project_date : {false, true}) {
+ SCOPED_TRACE(project_date);
+ auto reader = create_reader_for_path(path);
+ TQueryOptions options;
+ options.__set_enable_orc_lazy_mat(false);
+ RuntimeState state {options, TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->predicate_columns = {field_projection(0)};
+ request->non_predicate_columns = {project_date ? field_projection(1)
+ : struct_child_projection(1, 0)};
+ request->conjuncts = {
+ VExprContext::create_shared(std::make_shared<NullableInt32GreaterThanExpr>(0, 7))};
+ ASSERT_TRUE(reader->open(request).ok());
+ // Only a decoded DATE can require disabling SARG to preserve a conversion error.
+ EXPECT_EQ(reader->get_total_rows(), project_date ? 3 : 0);
+ EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, project_date ? 0 : 1);
+ if (project_date) {
+ auto block = build_file_block(schema);
+ size_t rows = 0;
+ bool eof = false;
+ const auto status = reader->get_block(&block, &rows, &eof);
+ EXPECT_TRUE(status.is<ErrorCode::DATA_QUALITY_ERROR>()) << status;
+ }
+ }
+}
+
+TEST_F(NewOrcReaderTest, DateSargRetainsUnsafeStripeWithoutDisablingSafeStripePruning) {
+ const auto path = (_test_dir / "mixed_date_stripes.orc").string();
+ auto type = std::unique_ptr<::orc::Type>(
+ ::orc::Type::buildTypeFromString("struct<id:int,d:date,payload:string>"));
+ MemoryOutputStream stream(4 * 1024 * 1024);
+ ::orc::WriterOptions writer_options;
+ writer_options.setStripeSize(1);
+ writer_options.setCompression(::orc::CompressionKind_NONE);
+ writer_options.setDictionaryKeySizeThreshold(0);
+ auto writer = ::orc::createWriter(*type, &stream, writer_options);
+ // The safe stripes cannot match id > 7, but the invalid middle stripe must be decoded.
+ for (const int64_t day : {0, -719469, 1}) {
+ auto batch = writer->createRowBatch(200);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto& ids = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[0]);
+ auto& dates = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[1]);
+ auto& payload = dynamic_cast<::orc::StringVectorBatch&>(*root.fields[2]);
+ std::vector<std::string> values;
+ values.reserve(200);
+ root.numElements = ids.numElements = dates.numElements = payload.numElements = 200;
+ for (int row = 0; row < 200; ++row) {
+ ids.data[row] = 0;
+ dates.data[row] = day;
+ values.emplace_back(2048, static_cast<char>('a' + row % 26));
+ set_string_value(payload, row, values.back());
+ }
+ writer->add(*batch);
+ }
+ writer->close();
+ std::ofstream out(path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ out.close();
+ ASSERT_EQ(get_orc_stripe_count(path), 3);
+
+ for (bool pruning : {false, true}) {
+ SCOPED_TRACE(pruning);
+ auto reader = create_reader_for_path(path);
+ TQueryOptions options;
+ options.__set_enable_orc_filter_by_min_max(pruning);
+ options.__set_enable_orc_lazy_mat(false);
+ RuntimeState state {options, TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->predicate_columns = {field_projection(0)};
+ request->non_predicate_columns = {field_projection(1)};
+ request->conjuncts = {
+ VExprContext::create_shared(std::make_shared<NullableInt32GreaterThanExpr>(0, 7))};
+ ASSERT_TRUE(reader->open(request).ok());
+ EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, pruning ? 2 : 0);
+ auto block = build_file_block({schema[0], schema[1]});
+ bool eof = false;
+ Status status;
+ while (!eof && status.ok()) {
+ block.clear_column_data();
+ size_t rows = 0;
+ status = reader->get_block(&block, &rows, &eof);
+ }
+ EXPECT_TRUE(status.is<ErrorCode::DATA_QUALITY_ERROR>()) << status;
+ EXPECT_NE(status.to_string().find("-719469"), std::string::npos);
+ }
+}
+
+TEST_F(NewOrcReaderTest, ReadExternalDateRejectsLargeOrdinal) {
+ // This external file encoded YYYYMMDD as an ORC day ordinal; the old dictionary hid it
+ // by substituting 1900-01-01. Reading other columns must remain possible.
+ const auto path = find_repo_file("be/test/exec/test_data/orc_scanner/date_large_ordinal.orc");
+ ASSERT_TRUE(std::filesystem::exists(path));
+ for (bool project_date : {false, true}) {
+ SCOPED_TRACE(project_date);
+ auto reader = create_reader_for_path(path.string());
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ ASSERT_EQ(schema.size(), 9);
+ ASSERT_EQ(remove_nullable(schema[3].type)->get_primitive_type(), TYPE_DATEV2);
+ const int column_id = project_date ? 3 : 0;
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(column_id)};
+ ASSERT_TRUE(reader->open(request).ok());
+ auto block = build_file_block({schema[column_id]});
+ size_t rows = 0;
+ bool eof = false;
+ const auto status = reader->get_block(&block, &rows, &eof);
+ if (project_date) {
+ EXPECT_TRUE(status.is<ErrorCode::DATA_QUALITY_ERROR>()) << status;
+ EXPECT_NE(
+ status.to_string().find("DATE value 20191111 is outside the Doris DATE range"),
+ std::string::npos);
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(rows, 10);
+ }
+ }
+}
+
+TEST_F(NewOrcReaderTest, ReadExternalMapDateRejectsOutOfRangeOrdinal) {
+ // Keep the external LZ4 fixture: the old offset dictionary silently replaced -719530
+ // with 1900-01-01, hiding an unrepresentable DATE inside the map values.
+ const auto path = find_repo_file("be/test/exec/test_data/orc_scanner/date_out_of_range.orc");
+ ASSERT_TRUE(std::filesystem::exists(path));
+ auto reader = create_reader_for_path(path.string());
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ ASSERT_EQ(schema.size(), 2);
+ ASSERT_EQ(remove_nullable(schema[1].type)->get_primitive_type(), TYPE_MAP);
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(0), field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+ auto block = build_file_block(schema);
+ size_t rows = 0;
+ bool eof = false;
+ const auto status = reader->get_block(&block, &rows, &eof);
+ EXPECT_TRUE(status.is<ErrorCode::DATA_QUALITY_ERROR>()) << status;
+ EXPECT_NE(status.to_string().find("DATE value -719530 is outside the Doris DATE range"),
+ std::string::npos);
+}
+
+TEST_F(NewOrcReaderTest, DatePreflightDoesNotReadPrunedStripeIndexes) {
+ std::array<size_t, 2> io_calls {};
+ std::array<size_t, 2> io_bytes {};
+ for (bool project_date : {false, true}) {
+ RuntimeProfile profile("date_preflight");
+ const auto path = find_repo_file("be/test/exec/test_data/orc_scanner/date_preflight.orc");
+ io::FileReaderStats stats;
+ auto io_ctx = std::make_shared<io::IOContext>();
+ io_ctx->file_reader_stats = &stats;
+ auto reader = create_reader_for_path(path.string(), &profile, io_ctx);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->predicate_columns = {field_projection(0)};
+ if (project_date) {
+ request->non_predicate_columns = {field_projection(1)};
+ }
+ request->conjuncts = {
+ VExprContext::create_shared(std::make_shared<NullableInt32GreaterThanExpr>(0, 7))};
+ ASSERT_TRUE(reader->open(request).ok());
+ EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 1);
+ ASSERT_TRUE(reader->close().ok());
+ io_calls[project_date] = stats.read_calls;
+ io_bytes[project_date] = stats.read_bytes;
+ }
+ // Projecting DATE must only inspect the already loaded file-level stripe metadata.
+ EXPECT_GT(io_calls[0], 0);
+ EXPECT_EQ(io_calls[1], io_calls[0]);
+ EXPECT_EQ(io_bytes[1], io_bytes[0]);
+}
+
+TEST_F(NewOrcReaderTest, DatePreflightSkipsMalformedIndexInPrunedStripe) {
+ // Only the unused column's ROW_INDEX stream ID is changed from 3 to 100.
+ const auto path =
+ find_repo_file("be/test/exec/test_data/orc_scanner/date_preflight_bad_index.orc");
+ auto reader = create_reader_for_path(path.string());
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->predicate_columns = {field_projection(0)};
+ request->non_predicate_columns = {field_projection(1)};
+ request->conjuncts = {
+ VExprContext::create_shared(std::make_shared<NullableInt32GreaterThanExpr>(0, 7))};
+ ASSERT_TRUE(reader->open(request).ok());
+ EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 1);
+}
+
+TEST_F(NewOrcReaderTest, DateCountRejectsTruncatedNestedStatistics) {
+ // The schema includes s.d (ID 3), but stripe statistics stop at its parent (ID 2).
+ const auto path =
+ find_repo_file("be/test/exec/test_data/orc_scanner/date_count_truncated_stats.orc");
+ auto reader = create_reader_for_path(path.string());
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(1)};
+ ASSERT_TRUE(reader->open(request).ok());
+ format::FileAggregateRequest aggregate_request;
+ aggregate_request.agg_type = TPushAggOp::type::COUNT;
+ aggregate_request.columns.push_back(
+ {.projection = format::LocalColumnIndex::top_level(format::LocalColumnId(1))});
+ format::FileAggregateResult result;
+ const auto status = reader->get_aggregate_result(aggregate_request, &result);
+ EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) << status;
+ // Falling back still reads both struct values from the intact data streams.
+ auto block = build_file_block({schema[1]});
+ size_t rows = 0;
+ bool eof = false;
+ ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
+ EXPECT_EQ(rows, 2);
+}
+
} // namespace
} // namespace doris
diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp
index a5f45839428..170800d6a95 100644
--- a/be/test/format_v2/parquet/parquet_reader_test.cpp
+++ b/be/test/format_v2/parquet/parquet_reader_test.cpp
@@ -5590,5 +5590,253 @@ TEST_F(NewParquetReaderTest, RowPositionReaderUsesFileLocalPositionsForScanRange
}
}
+class YearZeroDatePredicate final : public VExpr {
+public:
+ explicit YearZeroDatePredicate(bool is_null)
+ : VExpr(std::make_shared<DataTypeUInt8>(), false), _is_null(is_null) {}
+ const std::string& expr_name() const override { return _name; }
+ void collect_slot_column_ids(std::set<int>& ids) const override { ids.insert(0); }
+ bool can_evaluate_zonemap_filter() const override { return true; }
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override {
+ auto map = ctx.zone_map(0);
+ if (map == nullptr) {
+ return ZoneMapFilterResult::kMayMatch;
+ }
+ if (_is_null) {
+ return map->has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ DateV2Value<DateV2ValueType> bound;
+ bound.unchecked_set_time(1, 1, 1, 0, 0, 0, 0);
+ return map->has_not_null && map->max_value >= Field::create_field<TYPE_DATEV2>(bound)
+ ? ZoneMapFilterResult::kMayMatch
+ : ZoneMapFilterResult::kNoMatch;
+ }
+ Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector,
+ size_t count, ColumnPtr& result) const override {
+ const auto& column = assert_cast<const ColumnNullable&>(*block->get_by_position(0).column);
+ const auto& dates = assert_cast<const ColumnDateV2&>(column.get_nested_column());
+ auto out = ColumnUInt8::create();
+ for (size_t i = 0; i < count; ++i) {
+ const auto row = selector == nullptr ? i : (*selector)[i];
+ out->insert_value(_is_null ? column.is_null_at(row)
+ : !column.is_null_at(row) &&
+ dates.get_data()[row].year() >= 1);
+ }
+ result = std::move(out);
+ return Status::OK();
+ }
+
+private:
+ bool _is_null;
+ const std::string _name = "YearZeroDatePredicate";
+};
+
+TEST_F(NewParquetReaderTest, DateInteriorLeapDayRequiresMaterializedAggregates) {
+ arrow::Date32Builder dates;
+ ASSERT_TRUE(dates.AppendValues({-719528, -719469, -719468}).ok());
+ std::shared_ptr<arrow::Array> array;
+ ASSERT_TRUE(dates.Finish(&array).ok());
+ const auto table =
+ arrow::Table::Make(arrow::schema({arrow::field("d", arrow::date32())}), {array});
+ auto output = arrow::io::FileOutputStream::Open(_file_path).ValueOrDie();
+ ::parquet::WriterProperties::Builder properties;
+ properties.disable_dictionary();
+ ASSERT_TRUE(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), output, 3,
+ properties.build())
+ .ok());
+ ASSERT_TRUE(output->Close().ok());
+
+ for (bool strict : {false, true}) {
+ for (bool pruning : {false, true}) {
+ auto filtered_reader = create_reader();
+ TQueryOptions options;
+ options.__set_enable_strict_cast(strict);
+ RuntimeState filtered_state {options, TQueryGlobals()};
+ ASSERT_TRUE(filtered_reader->init(&filtered_state).ok());
+ std::vector<format::ColumnDefinition> filtered_schema;
+ ASSERT_TRUE(filtered_reader->get_schema(&filtered_schema).ok());
+ auto filtered_request = std::make_shared<format::FileScanRequest>();
+ filtered_request->predicate_columns = {
+ format::LocalColumnIndex::top_level(format::LocalColumnId(0))};
+ filtered_request->conjuncts = {
+ VExprContext::create_shared(std::make_shared<YearZeroDatePredicate>(!strict))};
+ filtered_request->metadata_pruning_safe_conjunct_count = pruning ? 1 : 0;
+ ASSERT_TRUE(filtered_reader->open(filtered_request).ok());
+ auto filtered_block = build_file_block(filtered_schema);
+ size_t filtered_rows = 0;
+ bool filtered_eof = false;
+ const auto status =
+ filtered_reader->get_block(&filtered_block, &filtered_rows, &filtered_eof);
+ if (strict) {
+ EXPECT_FALSE(status.ok()) << "pruning=" << pruning;
+ EXPECT_NE(status.to_string().find("-719469"), std::string::npos);
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(filtered_rows, 1) << "pruning=" << pruning;
+ if (filtered_rows == 1) {
+ EXPECT_TRUE(filtered_block.get_by_position(0).column->is_null_at(0));
+ }
+ }
+ }
+ }
+ auto reader = create_reader();
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {
+ format::LocalColumnIndex::top_level(format::LocalColumnId(0))};
+ ASSERT_TRUE(reader->open(request).ok());
+ for (auto operation : {TPushAggOp::type::COUNT, TPushAggOp::type::MINMAX}) {
+ format::FileAggregateRequest aggregate;
+ aggregate.agg_type = operation;
+ aggregate.columns.push_back({.projection = request->non_predicate_columns[0]});
+ format::FileAggregateResult result;
+ EXPECT_TRUE(reader->get_aggregate_result(aggregate, &result)
+ .is<ErrorCode::NOT_IMPLEMENTED_ERROR>());
+ }
+ auto block = build_file_block(schema);
+ size_t rows = 0;
+ bool eof = false;
+ ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
+ ASSERT_EQ(rows, 3);
+ const auto& values = assert_cast<const ColumnNullable&>(*block.get_by_position(0).column);
+ EXPECT_FALSE(values.is_null_at(0));
+ EXPECT_TRUE(values.is_null_at(1));
+ EXPECT_FALSE(values.is_null_at(2));
+}
+
+TEST_F(NewParquetReaderTest, DateCountKeepsSafeMetadataShortcut) {
+ const std::vector<std::vector<std::optional<int32_t>>> cases = {
+ {-719528, -719470}, {-719468, 19723}, {std::nullopt, std::nullopt}};
+ for (const auto& values : cases) {
+ for (bool nullable : {false, true}) {
+ if (!nullable && !values[0].has_value()) {
+ continue;
+ }
+ arrow::Date32Builder dates;
+ for (auto value : values) {
+ ASSERT_TRUE((value.has_value() ? dates.Append(*value) : dates.AppendNull()).ok());
+ }
+ std::shared_ptr<arrow::Array> array;
+ ASSERT_TRUE(dates.Finish(&array).ok());
+ auto table = arrow::Table::Make(
+ arrow::schema({arrow::field("d", arrow::date32(), nullable)}), {array});
+ auto output = arrow::io::FileOutputStream::Open(_file_path).ValueOrDie();
+ ASSERT_TRUE(
+ ::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), output, 2)
+ .ok());
+ ASSERT_TRUE(output->Close().ok());
+ auto reader = create_reader();
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ ASSERT_TRUE(reader->open(std::make_shared<format::FileScanRequest>()).ok());
+ format::FileAggregateRequest request;
+ request.agg_type = TPushAggOp::type::COUNT;
+ request.columns.push_back(
+ {.projection = format::LocalColumnIndex::top_level(format::LocalColumnId(0))});
+ format::FileAggregateResult result;
+ ASSERT_TRUE(reader->get_aggregate_result(request, &result).ok());
+ EXPECT_EQ(result.count, values[0].has_value() ? 2 : 0);
+ }
+ }
+}
+
+TEST_F(NewParquetReaderTest, NestedDateCountPreservesStrictConversionErrors) {
+ for (bool invalid : {false, true}) {
+ for (bool all_null : {false, true}) {
+ if (invalid && all_null) {
+ continue;
+ }
+ auto dates = std::make_shared<arrow::Date32Builder>();
+ arrow::ListBuilder lists(arrow::default_memory_pool(), dates);
+ for (int32_t day : {-719528, invalid ? -719469 : -719470, -719470}) {
+ ASSERT_TRUE(lists.Append().ok());
+ ASSERT_TRUE((all_null ? dates->AppendNull() : dates->Append(day)).ok());
+ }
+ const auto list = finish_array(&lists);
+ const auto values = std::static_pointer_cast<arrow::ListArray>(list)->values();
+ const auto record = arrow::StructArray::Make({values, build_int32_array({0, 1, 2})},
+ {arrow::field("d", arrow::date32()),
+ arrow::field("n", arrow::int32())})
+ .ValueOrDie();
+ const auto offsets = build_int32_array({0, 1, 2, 3});
+ const auto map =
+ arrow::MapArray::FromArrays(offsets, build_int32_array({0, 1, 2}), values)
+ .ValueOrDie();
+ const auto table = arrow::Table::Make(arrow::schema({arrow::field("a", list->type()),
+ arrow::field("s", record->type()),
+ arrow::field("m", map->type())}),
+ {list, record, map});
+ auto output = arrow::io::FileOutputStream::Open(_file_path).ValueOrDie();
+ ::parquet::WriterProperties::Builder properties;
+ properties.disable_dictionary();
+ ASSERT_TRUE(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), output,
+ 3, properties.build())
+ .ok());
+ ASSERT_TRUE(output->Close().ok());
+ for (bool strict : {false, true}) {
+ for (int column_id = 0; column_id < 3; ++column_id) {
+ SCOPED_TRACE(testing::Message()
+ << "invalid=" << invalid << " all_null=" << all_null
+ << " strict=" << strict << " column=" << column_id);
+ auto reader = create_reader();
+ TQueryOptions options;
+ options.__set_enable_strict_cast(strict);
+ RuntimeState state {options, TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ const auto projection =
+ format::LocalColumnIndex::top_level(format::LocalColumnId(column_id));
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {projection};
+ ASSERT_TRUE(reader->open(request).ok());
+ format::FileAggregateRequest aggregate;
+ aggregate.agg_type = TPushAggOp::type::COUNT;
+ aggregate.columns.push_back({.projection = projection});
+ format::FileAggregateResult result;
+ const auto status = reader->get_aggregate_result(aggregate, &result);
+ if (invalid && strict) {
+ EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) << status;
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(result.count, 3);
+ }
+ auto block = build_file_block(schema);
+ size_t rows = 0;
+ bool eof = false;
+ const auto scan_status = reader->get_block(&block, &rows, &eof);
+ if (invalid && strict) {
+ EXPECT_TRUE(scan_status.is<ErrorCode::DATA_QUALITY_ERROR>()) << scan_status;
+ EXPECT_NE(scan_status.to_string().find("-719469"), std::string::npos);
+ } else {
+ ASSERT_TRUE(scan_status.ok()) << scan_status;
+ EXPECT_EQ(rows, 3);
+ }
+ }
+ }
+ // An unprojected DATE sibling must not disable a safe shape-only COUNT.
+ auto reader = create_reader();
+ TQueryOptions options;
+ options.__set_enable_strict_cast(true);
+ RuntimeState state {options, TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ auto projection = format::LocalColumnIndex::partial_local(1);
+ projection.children.push_back(format::LocalColumnIndex::local(1));
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {projection};
+ ASSERT_TRUE(reader->open(request).ok());
+ format::FileAggregateRequest aggregate;
+ aggregate.agg_type = TPushAggOp::type::COUNT;
+ aggregate.columns.push_back({.projection = projection});
+ format::FileAggregateResult result;
+ ASSERT_TRUE(reader->get_aggregate_result(aggregate, &result).ok());
+ EXPECT_EQ(result.count, 3);
+ }
+ }
+}
+
} // namespace
} // namespace doris
diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp
index 6f51499bc33..b06a72f0977 100644
--- a/be/test/format_v2/parquet/parquet_statistics_test.cpp
+++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp
@@ -28,6 +28,7 @@
#include <optional>
#include <set>
#include <string>
+#include <tuple>
#include <unordered_map>
#include <utility>
#include <vector>
@@ -2212,5 +2213,178 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering)
EXPECT_EQ(selected_ranges[0].length, 100);
}
+TEST(NativeParquetStatisticsTest, DateInteriorLeapDayInvalidatesPhysicalNullStatistics) {
+ format::parquet::ParquetColumnSchema schema;
+ schema.type = make_nullable(std::make_shared<DataTypeDateV2>());
+ schema.type_descriptor.doris_type = schema.type;
+ schema.type_descriptor.physical_type = tparquet::Type::INT32;
+ auto encode = [](int32_t value) {
+ return std::string(reinterpret_cast<const char*>(&value), sizeof(value));
+ };
+ for (const auto& [min, max, safe] :
+ std::vector<std::tuple<int32_t, int32_t, bool>> {{-719528, -719468, false},
+ {-719469, -719469, false},
+ {-719528, -719470, true},
+ {-719468, 19723, true}}) {
+ tparquet::Statistics stats;
+ stats.__set_null_count(0);
+ stats.__set_min_value(encode(min));
+ stats.__set_max_value(encode(max));
+ const auto decoded = format::parquet::ParquetStatisticsUtils::TransformColumnStatistics(
+ schema, &stats, 3, nullptr);
+ EXPECT_EQ(safe, decoded.has_min_max);
+ EXPECT_EQ(safe, decoded.has_null_count);
+ }
+}
+
+class DateStatisticsPredicate final : public VExpr {
+public:
+ explicit DateStatisticsPredicate(bool is_null, int slot = 0)
+ : VExpr(std::make_shared<DataTypeUInt8>(), false), _is_null(is_null), _slot(slot) {}
+ const std::string& expr_name() const override { return _name; }
+ Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t,
+ ColumnPtr&) const override {
+ return Status::InternalError("Metadata-only predicate");
+ }
+ bool can_evaluate_zonemap_filter() const override { return true; }
+ void collect_slot_column_ids(std::set<int>& ids) const override { ids.insert(_slot); }
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override {
+ const auto map = ctx.zone_map(_slot);
+ if (map == nullptr) {
+ return ZoneMapFilterResult::kMayMatch;
+ }
+ if (_is_null) {
+ return map->has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ DateV2Value<DateV2ValueType> bound;
+ bound.unchecked_set_time(1, 1, 1, 0, 0, 0, 0);
+ return map->has_not_null && map->max_value >= Field::create_field<TYPE_DATEV2>(bound)
+ ? ZoneMapFilterResult::kMayMatch
+ : ZoneMapFilterResult::kNoMatch;
+ }
+
+private:
+ bool _is_null;
+ int _slot;
+ const std::string _name = "DateStatisticsPredicate";
+};
+
+TEST(NativeParquetStatisticsTest, UnsafeDatePageDoesNotDisableSafePagePruning) {
+ std::vector<std::unique_ptr<format::parquet::ParquetColumnSchema>> schema;
+ format::FileScanRequest request;
+ for (int slot = 0; slot < 2; ++slot) {
+ auto column = std::make_unique<format::parquet::ParquetColumnSchema>();
+ column->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE;
+ column->local_id = column->leaf_column_id = slot;
+ column->type = make_nullable(std::make_shared<DataTypeDateV2>());
+ column->type_descriptor.doris_type = column->type;
+ column->type_descriptor.physical_type = tparquet::Type::INT32;
+ schema.push_back(std::move(column));
+ request.local_positions.emplace(format::LocalColumnId(slot), format::LocalIndex(slot));
+ request.predicate_columns.push_back(format::LocalColumnIndex::local(slot));
+ }
+ auto encode = [](int32_t value) {
+ return std::string(reinterpret_cast<const char*>(&value), sizeof(value));
+ };
+ format::parquet::NativeParquetPageIndex index;
+ index.column_index.__set_min_values({encode(-719528), encode(-719528), encode(19723)});
+ index.column_index.__set_max_values({encode(-719470), encode(-719468), encode(19724)});
+ index.column_index.__set_null_pages({false, false, false});
+ index.column_index.__set_null_counts({0, 0, 0});
+ for (int page = 0; page < 3; ++page) {
+ tparquet::PageLocation location;
+ location.__set_offset(page * 10);
+ location.__set_compressed_page_size(10);
+ location.__set_first_row_index(page * 3);
+ index.offset_index.page_locations.push_back(location);
+ }
+ std::unordered_map<int, format::parquet::NativeParquetPageIndex> indexes;
+ indexes.emplace(0, index);
+ indexes.emplace(1, index);
+ tparquet::ColumnOrder order;
+ order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder());
+ tparquet::FileMetaData metadata;
+ metadata.__set_column_orders({order, order});
+ tparquet::RowGroup group;
+ group.__set_num_rows(9);
+ for (bool is_null : {false, true}) {
+ for (bool compound : {false, true}) {
+ SCOPED_TRACE(testing::Message() << "is_null=" << is_null << " compound=" << compound);
+ VExprSPtr predicate = std::make_shared<DateStatisticsPredicate>(is_null);
+ if (compound) {
+ TExprNode node;
+ node.__set_node_type(TExprNodeType::COMPOUND_PRED);
+ node.__set_opcode(TExprOpcode::COMPOUND_OR);
+ node.__set_type(std::make_shared<DataTypeUInt8>()->to_thrift());
+ node.__set_num_children(2);
+ node.__set_is_nullable(false);
+ auto either = VCompoundPred::create_shared(node);
+ either->add_child(predicate);
+ either->add_child(std::make_shared<DateStatisticsPredicate>(is_null, 1));
+ predicate = either;
+ }
+ request.conjuncts = {VExprContext::create_shared(predicate)};
+ std::vector<format::parquet::RowRange> ranges;
+ std::map<int, format::parquet::ParquetPageSkipPlan> plans;
+ ASSERT_TRUE(
+ format::parquet::select_row_group_ranges_by_native_page_index(
+ metadata, group, indexes, schema, request, 9, &ranges, &plans, nullptr)
+ .ok());
+ ASSERT_EQ(ranges.size(), 1);
+ EXPECT_EQ(ranges[0].start, 3);
+ EXPECT_EQ(ranges[0].length, is_null ? 3 : 6);
+ }
+ }
+}
+
+TEST(NativeParquetStatisticsTest, DateInteriorLeapDayKeepsPageCandidates) {
+ auto column = std::make_unique<format::parquet::ParquetColumnSchema>();
+ column->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE;
+ column->local_id = 0;
+ column->leaf_column_id = 0;
+ column->type = make_nullable(std::make_shared<DataTypeDateV2>());
+ column->type_descriptor.doris_type = column->type;
+ column->type_descriptor.physical_type = tparquet::Type::INT32;
+ std::vector<std::unique_ptr<format::parquet::ParquetColumnSchema>> schema;
+ schema.push_back(std::move(column));
+ auto encode = [](int32_t value) {
+ return std::string(reinterpret_cast<const char*>(&value), sizeof(value));
+ };
+ format::parquet::NativeParquetPageIndex index;
+ index.column_index.__set_min_values({encode(-719528)});
+ index.column_index.__set_max_values({encode(-719468)});
+ index.column_index.__set_null_pages({false});
+ index.column_index.__set_null_counts({0});
+ tparquet::PageLocation location;
+ location.__set_offset(0);
+ location.__set_compressed_page_size(10);
+ location.__set_first_row_index(0);
+ index.offset_index.__set_page_locations({location});
+ std::unordered_map<int, format::parquet::NativeParquetPageIndex> indexes;
+ indexes.emplace(0, std::move(index));
+ tparquet::ColumnOrder order;
+ order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder());
+ tparquet::FileMetaData metadata;
+ metadata.__set_column_orders({order});
+ tparquet::RowGroup group;
+ group.__set_num_rows(3);
+ for (bool is_null : {false, true}) {
+ format::FileScanRequest request;
+ request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0));
+ request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0))};
+ request.conjuncts = {
+ VExprContext::create_shared(std::make_shared<DateStatisticsPredicate>(is_null))};
+ std::vector<format::parquet::RowRange> ranges;
+ std::map<int, format::parquet::ParquetPageSkipPlan> plans;
+ ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index(
+ metadata, group, indexes, schema, request, 3, &ranges, &plans, nullptr)
+ .ok());
+ EXPECT_EQ(ranges.size(), 1);
+ if (ranges.size() == 1) {
+ EXPECT_EQ(ranges[0].length, 3);
+ }
+ }
+}
+
} // namespace
} // namespace doris
diff --git a/regression-test/data/arrow_flight_sql_p0/test_date_year_zero.out b/regression-test/data/arrow_flight_sql_p0/test_date_year_zero.out
new file mode 100644
index 00000000000..f338898c9cb
--- /dev/null
+++ b/regression-test/data/arrow_flight_sql_p0/test_date_year_zero.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !source --
+1 0000-01-01 0000-01-01
+2 0000-02-28 \N
+3 0000-03-01 0000-03-01
+4 0001-01-01 0001-01-01
+5 1969-12-31 1969-12-31
+6 1970-01-01 1970-01-01
+7 2024-01-01 2024-01-01
+8 9999-12-31 9999-12-31
+
diff --git a/regression-test/data/export_p0/outfile/test_outfile_date_year_zero.out b/regression-test/data/export_p0/outfile/test_outfile_date_year_zero.out
new file mode 100644
index 00000000000..6c1d9003d9d
--- /dev/null
+++ b/regression-test/data/export_p0/outfile/test_outfile_date_year_zero.out
@@ -0,0 +1,37 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !source --
+1 0000-01-01
+10 \N
+2 0000-01-31
+3 0000-02-28
+4 0000-03-01
+5 0001-01-01
+6 1969-12-31
+7 1970-01-01
+8 2024-01-01
+9 9999-12-31
+
+-- !readback_parquet --
+1 0000-01-01
+10 \N
+2 0000-01-31
+3 0000-02-28
+4 0000-03-01
+5 0001-01-01
+6 1969-12-31
+7 1970-01-01
+8 2024-01-01
+9 9999-12-31
+
+-- !readback_orc --
+1 0000-01-01
+10 \N
+2 0000-01-31
+3 0000-02-28
+4 0000-03-01
+5 0001-01-01
+6 1969-12-31
+7 1970-01-01
+8 2024-01-01
+9 9999-12-31
+
diff --git a/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out b/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out
index dcf3cdb63ef..f31f2a50d2f 100644
--- a/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out
+++ b/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out
@@ -209,28 +209,6 @@
-- !test_15 --
1 0 0 1 apple 3.5
--- !test_16 --
-1 {-99999.99999:"1900-01-01"}
-1 {-99999.99999:"1900-01-01"}
-10 {null:"1000-10-16"}
-10 {null:"1000-10-16"}
-2 {99999.99999:"9999-12-31"}
-2 {99999.99999:"9999-12-31"}
-3 {0.00000:null}
-3 {0.00000:null}
-4 {null:"0000-12-29"}
-4 {null:"0000-12-29"}
-5 {null:null}
-5 {null:null}
-6 {-101.00000:null}
-6 {-101.00000:null}
-7 {null:null}
-7 {null:null}
-8 {0.00001:"2022-03-02"}
-8 {0.00001:"2022-03-02"}
-9 {null:null}
-9 {null:null}
-
-- !test_17 --
10 hello 10 10.01 \N
20 world 20 20.01 \N
@@ -247,18 +225,6 @@
-- !test_20 --
2022-04-09T07:13 2022-04-09
--- !test_21 --
-0 \N doris \N 1.567 1.567 12345 1 doris
-1 true doris 1900-01-01 1.567 1.567 12345 1 doris
-2 \N doris 1900-01-01 1.567 1.567 12345 1 doris
-3 true doris 1900-01-01 1.567 1.567 12345 1 doris
-4 \N doris \N 1.567 1.567 12345 1 doris
-5 true doris 1900-01-01 1.567 1.567 12345 1 doris
-6 \N doris 1900-01-01 1.567 1.567 12345 1 doris
-7 true doris 1900-01-01 1.567 1.567 12345 1 doris
-8 \N doris \N 1.567 1.567 12345 1 doris
-9 true doris 1900-01-01 1.567 1.567 12345 1 doris
-
-- !test_22 --
-128
diff --git a/regression-test/suites/arrow_flight_sql_p0/test_date_year_zero.groovy b/regression-test/suites/arrow_flight_sql_p0/test_date_year_zero.groovy
new file mode 100644
index 00000000000..c480a29e24b
--- /dev/null
+++ b/regression-test/suites/arrow_flight_sql_p0/test_date_year_zero.groovy
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Doris numbers DATE values in MySQL's calendar, where year 0 is not a leap year. Arrow `date32`
+// is days since 1970-01-01 in the proleptic Gregorian calendar, where year 0 IS a leap year, so
+// the two disagree for 0000-01-01 .. 0000-02-28. The MySQL protocol ships the year/month/day
+// fields directly and therefore never depends on a calendar; it is the control path here. Any
+// difference between the two protocols means the Arrow day ordinal is wrong.
+// See https://github.com/apache/doris/issues/67366
+suite("test_date_year_zero", "arrow_flight_sql") {
+ sql "DROP TABLE IF EXISTS test_date_year_zero"
+ sql """
+ CREATE TABLE test_date_year_zero (
+ id INT,
+ d DATE,
+ d_null DATE NULL,
+ arr ARRAY<DATE>
+ ) DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_num" = "1");
+ """
+ sql """
+ INSERT INTO test_date_year_zero VALUES
+ (1, '0000-01-01', '0000-01-01', ['0000-01-01', '0000-02-28']),
+ (2, '0000-02-28', NULL, ['0000-03-01']),
+ (3, '0000-03-01', '0000-03-01', ['0001-01-01']),
+ (4, '0001-01-01', '0001-01-01', ['1969-12-31', '1970-01-01']),
+ (5, '1969-12-31', '1969-12-31', ['2024-01-01']),
+ (6, '1970-01-01', '1970-01-01', ['9999-12-31']),
+ (7, '2024-01-01', '2024-01-01', ['0000-01-01']),
+ (8, '9999-12-31', '9999-12-31', NULL);
+ """
+
+ // The absolute anchor. The MySQL protocol carries year/month/day verbatim, so these strings
+ // are what the Arrow path below must reproduce. Cast to string: java.sql.Date runs year-zero
+ // values through the Julian/Gregorian hybrid calendar.
+ order_qt_source """
+ SELECT CAST(id AS STRING), CAST(d AS STRING), CAST(d_null AS STRING)
+ FROM test_date_year_zero
+ """
+
+ // Read through the driver's own rendering rather than getObject(), for the same reason.
+ def fetchStrings = { java.sql.Connection conn, String stmtSql ->
+ def rows = []
+ conn.prepareStatement(stmtSql).withCloseable { st ->
+ st.executeQuery().withCloseable { rs ->
+ def columnCount = rs.metaData.columnCount
+ while (rs.next()) {
+ def row = []
+ for (int i = 1; i <= columnCount; ++i) {
+ row.add(rs.getString(i))
+ }
+ rows.add(row)
+ }
+ }
+ }
+ return rows
+ }
+
+ def query = "SELECT id, d, d_null FROM test_date_year_zero ORDER BY id"
+ def viaMysql = fetchStrings(context.getConnection(), query)
+ def viaFlight = fetchStrings(context.getArrowFlightSqlConnection(),
+ "USE ${context.dbName};" + query)
+ assertEquals(8, viaMysql.size())
+ // Before the fix, Arrow reported 0000-01-02 for row 1 and 0000-02-29 for row 2.
+ assertEquals(viaMysql, viaFlight,
+ "the Arrow and MySQL protocols disagree on the DATE values")
+
+ // ARRAY<DATE> has no encoding of its own, it delegates each element to the DATE SerDe. The
+ // Arrow Flight JDBC driver renders a list<date32> as its raw day counts rather than as dates,
+ // which makes this the strictest check available here: it pins the exact wire values.
+ // 0000-01-01 is -719528 and 0000-02-28 is -719470 in the proleptic Gregorian calendar; the
+ // pre-fix encoding produced -719527 and -719469.
+ def arrayQuery = "SELECT arr FROM test_date_year_zero WHERE id = 1"
+ def arrayViaFlight =
+ fetchStrings(context.getArrowFlightSqlConnection(),
+ "USE ${context.dbName};" + arrayQuery)[0][0].toString()
+ def arrayElements = arrayViaFlight.replaceAll(/[\[\]\s"]/, "").split(",") as List
+ assertEquals(["-719528", "-719470"], arrayElements,
+ "ARRAY<DATE> elements are not proleptic Gregorian day counts, got: ${arrayViaFlight}")
+}
diff --git a/regression-test/suites/export_p0/outfile/test_outfile_date_year_zero.groovy b/regression-test/suites/export_p0/outfile/test_outfile_date_year_zero.groovy
new file mode 100644
index 00000000000..7d4cfaa1a13
--- /dev/null
+++ b/regression-test/suites/export_p0/outfile/test_outfile_date_year_zero.groovy
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Parquet and ORC both store DATE as days since 1970-01-01 in the proleptic Gregorian calendar,
+// which differs from Doris's MySQL-calendar day number for 0000-01-01 .. 0000-02-28. The writer
+// (which shares the Arrow date32 encoder) and the reader must therefore agree, otherwise a table
+// exported and re-read by Doris comes back shifted by a day.
+// See https://github.com/apache/doris/issues/67366
+suite("test_outfile_date_year_zero", "p0") {
+ String ak = getS3AK()
+ String sk = getS3SK()
+ String s3_endpoint = getS3Endpoint()
+ String region = getS3Region()
+ String bucket = context.config.otherConfigs.get("s3BucketName")
+
+ def outFilePath = "${bucket}/outfile/date_year_zero/exp_"
+
+ sql """ DROP TABLE IF EXISTS test_outfile_date_year_zero_table """
+ sql """
+ CREATE TABLE test_outfile_date_year_zero_table (
+ `id` INT NOT NULL,
+ `d` DATE NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_num" = "1");
+ """
+ // 0000-01-01 .. 0000-02-28 are the window where the two calendars disagree; 0000-03-01 onwards
+ // they coincide, so keep dates on both sides of the boundary plus the two range limits.
+ sql """
+ INSERT INTO test_outfile_date_year_zero_table VALUES
+ (1, '0000-01-01'), (2, '0000-01-31'), (3, '0000-02-28'), (4, '0000-03-01'),
+ (5, '0001-01-01'), (6, '1969-12-31'), (7, '1970-01-01'), (8, '2024-01-01'),
+ (9, '9999-12-31'), (10, NULL);
+ """
+
+ // Cast to string: the MySQL JDBC driver cannot materialise a year-zero DATE as java.sql.Date.
+ order_qt_source """
+ SELECT CAST(id AS STRING), CAST(d AS STRING) FROM test_outfile_date_year_zero_table
+ """
+
+ def outfile_to_S3 = { format ->
+ def res = sql """
+ SELECT * FROM test_outfile_date_year_zero_table t
+ INTO OUTFILE "s3://${outFilePath}"
+ FORMAT AS ${format}
+ PROPERTIES (
+ "s3.endpoint" = "${s3_endpoint}",
+ "s3.region" = "${region}",
+ "s3.secret_key"="${sk}",
+ "s3.access_key" = "${ak}"
+ );
+ """
+ def outfile_url = res[0][3]
+ def uri = "http://${bucket}.${s3_endpoint}" +
+ outfile_url.substring(5 + bucket.length(), outfile_url.length() - 1) + "0." + format
+ return """
+ SELECT CAST(id AS STRING), CAST(d AS STRING) FROM S3 (
+ "uri" = "${uri}",
+ "ACCESS_KEY" = "${ak}",
+ "SECRET_KEY" = "${sk}",
+ "format" = "${format}",
+ "region" = "${region}"
+ )
+ """
+ }
+
+ // Both read-backs must return exactly the source rows: same values, same NULL.
+ order_qt_readback_parquet(outfile_to_S3("parquet"))
+ order_qt_readback_orc(outfile_to_S3("orc"))
+}
diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_date_year_zero.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_date_year_zero.groovy
new file mode 100644
index 00000000000..829353ae7f3
--- /dev/null
+++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_date_year_zero.groovy
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Partition metadata and Parquet DATE payloads must use the same Gregorian epoch days.
+suite("test_iceberg_write_date_year_zero",
+ "p0,external,iceberg,external_docker,external_docker_iceberg") {
+ if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest"))) {
+ logger.info("disable iceberg test")
+ return
+ }
+ def env = context.config.otherConfigs
+ def catalog = "test_iceberg_write_date_year_zero"
+ def database = "iceberg_write_date_year_zero_db"
+ sql "drop catalog if exists ${catalog}"
+ sql """create catalog ${catalog} properties (
+ "type" = "iceberg", "iceberg.catalog.type" = "rest",
+ "uri" = "http://${env.get('externalEnvIp')}:${env.get('iceberg_rest_uri_port')}",
+ "s3.access_key" = "admin", "s3.secret_key" = "password",
+ "s3.endpoint" = "http://${env.get('externalEnvIp')}:${env.get('iceberg_minio_port')}",
+ "s3.region" = "us-east-1", "meta.cache.iceberg.table.ttl-second" = "0",
+ "meta.cache.iceberg.schema.ttl-second" = "0")"""
+ sql "switch ${catalog}"
+ sql "drop database if exists ${database} force"
+ sql "create database ${database}"
+ sql "use ${database}"
+ try {
+ def dates = ['0000-01-01', '0000-02-28', '0000-03-01', '1969-12-31', '1970-01-01', '2024-01-01']
+ def rows = dates.withIndex().collect { d, i -> "(${i}, date '${d}', date '${d}')" }.join(', ')
+ rows += ', (6, NULL, NULL)'
+ for (def name : ['doris_dates', 'spark_dates']) {
+ sql """create table ${name} (id int, d_day date, d_bucket date)
+ partition by list (day(d_day), bucket(16, d_bucket)) ()
+ properties ("format-version" = "2", "write.format.default" = "parquet")"""
+ }
+ sql "insert into doris_dates values ${rows}"
+ spark_iceberg "insert into demo.${database}.spark_dates values ${rows}"
+ for (def name : ['doris_dates', 'spark_dates']) {
+ sql "refresh table ${name}"
+ spark_iceberg "refresh table demo.${database}.${name}"
+ def projection = 'cast(id as string), cast(d_day as string), cast(d_bucket as string)'
+ assertSparkDorisResultEquals(
+ spark_iceberg("select ${projection} from demo.${database}.${name} order by id"),
+ sql("select ${projection} from ${name} order by id"))
+ dates.eachWithIndex { d, i ->
+ // An unfiltered scan alone cannot detect partition metadata contradicting the file.
+ for (def field : ['d_day', 'd_bucket']) {
+ def predicate = "${field} = date '${d}'"
+ def actual = spark_iceberg("select id from demo.${database}.${name} where ${predicate}")
+ assertEquals([[i]].toString(), actual.toString())
+ assertSparkDorisResultEquals(actual, sql("select id from ${name} where ${predicate}"))
+ }
+ }
+ }
+ def partitionQuery = { name ->
+ spark_iceberg("""select cast(partition.d_day_day as string),
+ cast(partition.d_bucket_bucket as string)
+ from demo.${database}.${name}.partitions order by 1, 2""")
+ }
+ assertEquals(partitionQuery('spark_dates').toString(), partitionQuery('doris_dates').toString())
+ } finally {
+ sql "drop database if exists ${database} force"
+ sql "switch internal"
+ sql "drop catalog if exists ${catalog}"
+ }
+}
diff --git a/regression-test/suites/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.groovy b/regression-test/suites/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.groovy
index 0afe32746da..d4162027410 100644
--- a/regression-test/suites/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.groovy
+++ b/regression-test/suites/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.groovy
@@ -139,10 +139,21 @@ suite("test_hdfs_orc_group6_orc_files","external,hive,tvf,external_docker") {
uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_orc/group6/map_decimal_date.lz4.orc"
- order_qt_test_16 """ select * from HDFS(
- "uri" = "${uri}",
- "hadoop.username" = "${hdfsUserName}",
- "format" = "orc"); """
+ def scannerV2 = sql "show variables like 'enable_file_scanner_v2'"
+ try {
+ sql "set enable_file_scanner_v2 = true"
+ // The map contains ordinal -719530, below 0000-01-01. Returning 1900-01-01
+ // was an unchecked conversion failure, not a valid interpretation of the file.
+ test {
+ sql """ select * from HDFS(
+ "uri" = "${uri}",
+ "hadoop.username" = "${hdfsUserName}",
+ "format" = "orc"); """
+ exception "DATE value -719530 is outside the Doris DATE range"
+ }
+ } finally {
+ sql "set enable_file_scanner_v2 = ${scannerV2[0][1]}"
+ }
uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_orc/group6/string-2-double.orc"
@@ -174,10 +185,21 @@ suite("test_hdfs_orc_group6_orc_files","external,hive,tvf,external_docker") {
uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_orc/group6/non_vec_orc_scanner.orc"
- order_qt_test_21 """ select * from HDFS(
- "uri" = "${uri}",
- "hadoop.username" = "${hdfsUserName}",
- "format" = "orc"); """
+ scannerV2 = sql "show variables like 'enable_file_scanner_v2'"
+ try {
+ sql "set enable_file_scanner_v2 = true"
+ // ORC DATE stores epoch days, not YYYYMMDD. The raw ordinal 20191111 is
+ // out of range; the old 1900-01-01 result silently hid that invalid value.
+ test {
+ sql """ select * from HDFS(
+ "uri" = "${uri}",
+ "hadoop.username" = "${hdfsUserName}",
+ "format" = "orc"); """
+ exception "DATE value 20191111 is outside the Doris DATE range"
+ }
+ } finally {
+ sql "set enable_file_scanner_v2 = ${scannerV2[0][1]}"
+ }
uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_orc/group6/tinyint.orc"
diff --git a/tools/legacy_date_migration/README.md b/tools/legacy_date_migration/README.md
new file mode 100644
index 00000000000..7420e05d9f9
--- /dev/null
+++ b/tools/legacy_date_migration/README.md
@@ -0,0 +1,117 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migrating files with the former Doris DATE encoding
+
+The corrected DATE encoder writes proleptic Gregorian epoch days to ORC,
+Parquet and Arrow. The former Doris encoder used MySQL calendar day numbers
+minus 719528. For dates from `0000-01-01` through `0000-02-28`, the former value
+is one day too large. For example, those two endpoints were written as
+`-719527` and `-719469`, respectively; their correct values are `-719528` and
+`-719470`. Dates from `0000-03-01` onward are unchanged.
+
+A corrected reader cannot infer the intended date from an old ordinal:
+`-719527` also encodes the legitimate Gregorian date `0000-01-02`.
+Consequently, old files do not become compatible merely by upgrading the
+reader. Identify files using writer provenance, not their date values.
+
+## Explicit offline migration
+
+This tool requires Python 3.8 or later and PyArrow with ORC and Parquet support. It
+accepts only files **confirmed to have been produced with the former Doris
+DATE encoding for every DATE field**. Do not run it on files from other
+writers, mixed-origin files, or already migrated outputs. Running the
+conversion twice can silently shift dates again; the confirmation flag is
+an operator assertion, not automatic format detection. The lower-bound
+check catches some repeated conversions but cannot identify all of them.
+
+```bash
+python tools/legacy_date_migration/migrate_legacy_dates.py \
+ --input legacy.orc --output migrated.orc --format orc \
+ --confirm-legacy-doris-date-encoding
+
+python tools/legacy_date_migration/migrate_legacy_dates.py \
+ --input legacy.parquet --output migrated.parquet --format parquet \
+ --confirm-legacy-doris-date-encoding
+```
+
+The tool reads raw DATE32 integers and subtracts one only within the legacy
+window `[-719527, -719469]`. It preserves NULLs, nested ARRAY/STRUCT/MAP fields
+and other column values. It rejects DATE ordinals outside the legacy Doris
+range. Conversion uses Parquet batches of at most 65536 rows or one ORC
+stripe at a time; a large stripe or large nested values can still require
+substantial memory. It never modifies its input or overwrites an existing
+output. Output is published only after writing finishes, using an exclusive
+hard link on the destination filesystem; filesystems without hard-link
+support will fail without publishing output.
+
+Arrow field/schema metadata and nullability are retained during conversion.
+Parquet preserves this Arrow schema metadata. ORC has different metadata and
+type conventions and cannot round-trip arbitrary Arrow metadata or field
+nullability; the tool rewrites ORC data and schema supported by PyArrow,
+not an exact copy of all writer-specific metadata. Compression, row groups,
+stripes, statistics and indexes are regenerated. Unsupported types or
+writer failures abort without publishing a partially converted file.
+
+Keep the original files and record which files were converted. Query the
+new files with `enable_file_scanner_v2=true`, comparing row counts, NULLs,
+modern dates, and the year-zero boundary dates against authoritative source
+values. Only switch consumers after these checks. The converted files must
+be read by the corrected reader; enabling an old reader again is not a
+rollback strategy for the new encoding.
+
+## Optional legacy ORC read path
+
+For confirmed legacy ORC files, the legacy scanner can recover the original
+Doris dates before rewriting them through the corrected writer:
+
+```sql
+SET enable_file_scanner_v2 = false;
+-- Read only the confirmed legacy ORC source and write to a NEW output location.
+SELECT * FROM legacy_orc_source
+INTO OUTFILE 's3://example-bucket/migrated/date_export_'
+FORMAT AS ORC
+PROPERTIES (...);
+SET enable_file_scanner_v2 = true;
+-- Validate the newly written files against the original source values.
+```
+
+The storage properties and source relation above are placeholders. Scope
+the setting to a dedicated migration session. This is not a general
+compatibility switch: legacy Parquet reading does not reliably restore
+these dates, and a source containing both corrected and legacy files must
+not be switched wholesale. Prefer the explicit offline tool when provenance
+is known and the source format is Parquet.
+
+## Iceberg and other table formats
+
+Do not replace data files in place or edit their paths in existing metadata.
+Existing snapshots, partition values and statistics still describe the old
+files. Use migrated files as an independent source, then use the table
+format's writer to populate a new table (or a supported transactional rewrite)
+so that partition transforms and metadata are recomputed consistently.
+Validate the new table before moving consumers and retain the old table and
+snapshots for rollback. This utility migrates individual files; it does not
+perform an Iceberg commit or change existing snapshots.
+
+## Tests
+
+```bash
+python -m unittest discover -s tools/legacy_date_migration -v
+```
diff --git a/tools/legacy_date_migration/migrate_legacy_dates.py b/tools/legacy_date_migration/migrate_legacy_dates.py
new file mode 100644
index 00000000000..8554972c0ae
--- /dev/null
+++ b/tools/legacy_date_migration/migrate_legacy_dates.py
@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Rewrite confirmed legacy Doris DATE32 files into proleptic Gregorian encoding."""
+
+import argparse
+import os
+import tempfile
+from pathlib import Path
+
+import pyarrow as pa
+import pyarrow.compute as pc
+import pyarrow.orc as orc
+import pyarrow.parquet as parquet
+
+
+def convert_array(array, visible=None):
+ """Preserve Arrow structure and nulls while correcting legacy DATE32 ordinals."""
+ data_type = array.type
+ visible = array.is_valid() if visible is None else pc.and_(visible, array.is_valid())
+ if pa.types.is_date32(data_type):
+ # Children beneath a null container have no date value, regardless of their payload.
+ days = pc.if_else(visible, array.cast(pa.int32()), pa.scalar(None, pa.int32()))
+ bounds = pc.min_max(days).as_py()
+ if bounds['min'] is not None and (
+ bounds['min'] < -719527 or bounds['max'] > 2932896):
+ raise ValueError('DATE value outside the legacy Doris range')
+ # The old MySQL-calendar encoder is one day ahead only before March of year zero.
+ early = pc.and_(pc.greater_equal(days, -719527), pc.less_equal(days, -719469))
+ return pc.if_else(early, pc.subtract(days, pa.scalar(1, pa.int32())), days).cast(pa.date32())
+ if pa.types.is_struct(data_type):
+ return pa.StructArray.from_arrays(
+ [convert_array(array.field(i), visible) for i in range(data_type.num_fields)],
+ fields=list(data_type), mask=array.is_null())
+ if pa.types.is_list(data_type) or pa.types.is_large_list(data_type):
+ start, end = array.offsets[0].as_py(), array.offsets[-1].as_py()
+ raw_values = array.values.slice(start, end - start)
+ offsets = pc.subtract(array.offsets, start)
+ parents = pc.list_parent_indices(pa.LargeListArray.from_arrays(offsets, raw_values))
+ values = convert_array(raw_values, pc.take(visible, parents))
+ constructor = pa.LargeListArray if pa.types.is_large_list(data_type) else pa.ListArray
+ return constructor.from_arrays(offsets, values, type=data_type, mask=array.is_null())
+ if pa.types.is_fixed_size_list(data_type):
+ size = data_type.list_size
+ raw_values = array.values.slice(array.offset * size, len(array) * size)
+ parents = pc.list_parent_indices(pa.FixedSizeListArray.from_arrays(raw_values, size))
+ values = convert_array(raw_values, pc.take(visible, parents))
+ return pa.FixedSizeListArray.from_arrays(values, type=data_type, mask=array.is_null())
+ if pa.types.is_map(data_type):
+ start, end = array.offsets[0].as_py(), array.offsets[-1].as_py()
+ offsets = pc.subtract(array.offsets, start)
+ keys = array.keys.slice(start, end - start)
+ parents = pc.list_parent_indices(pa.LargeListArray.from_arrays(offsets, keys))
+ child_visible = pc.take(visible, parents)
+ converted_keys = convert_array(keys, child_visible)
+ if pa.types.is_date32(keys.type):
+ # Arrow requires non-null map keys even in an unreachable null map slot.
+ converted_keys = pc.fill_null(converted_keys, pa.scalar(0, pa.date32()))
+ return pa.MapArray.from_arrays(
+ offsets,
+ converted_keys,
+ convert_array(array.items.slice(start, end - start), child_visible),
+ type=data_type, mask=array.is_null())
+ if pa.types.is_dictionary(data_type):
+ used = pc.unique(pc.filter(array.indices, visible)).cast(pa.int64())
+ dictionary_visible = pc.is_in(pa.array(range(len(array.dictionary)), type=pa.int64()),
+ value_set=used)
+ return pa.DictionaryArray.from_arrays(
+ array.indices, convert_array(array.dictionary, dictionary_visible),
+ ordered=data_type.ordered)
+ if pa.types.is_union(data_type) or isinstance(data_type, pa.ExtensionType):
+ raise ValueError('Unsupported Arrow type; refusing a potentially partial migration: '
+ + str(data_type))
+ return array
+
+
+def convert_table(table):
+ """Convert all DATE32 fields, including nested fields and all chunks."""
+ columns = [pa.chunked_array([convert_array(chunk) for chunk in column.chunks],
+ type=column.type) for column in table.columns]
+ return pa.Table.from_arrays(columns, schema=table.schema)
+
+
+def migrate(source, destination, file_format, confirmed_legacy=False):
+ """Write a new file; source provenance must be confirmed by the caller."""
+ if not confirmed_legacy:
+ raise ValueError('Explicit confirmation of legacy Doris DATE encoding is required')
+ source, destination = Path(source), Path(destination)
+ if source.resolve() == destination.resolve():
+ raise ValueError('Source and destination must be different files')
+ if file_format not in ('orc', 'parquet'):
+ raise ValueError('Format must be orc or parquet')
+ # Convert one stripe/batch at a time to avoid loading the whole data file into memory.
+ reader = orc.ORCFile(source) if file_format == 'orc' else parquet.ParquetFile(source)
+ chunks = (reader.read_stripe(i) for i in range(reader.nstripes)) if file_format == 'orc' else (
+ reader.iter_batches(batch_size=65536))
+ schema = reader.schema if file_format == 'orc' else reader.schema_arrow
+ if destination.exists():
+ raise FileExistsError(destination)
+ # Publish with an exclusive hard link only after the writer succeeds, including its footer.
+ # An existence check alone races with another writer and could overwrite its output.
+ temporary = tempfile.NamedTemporaryFile(prefix='.legacy-date-', dir=destination.parent,
+ delete=False)
+ temporary_path = Path(temporary.name)
+ try:
+ with temporary:
+ writer = (orc.ORCWriter(temporary) if file_format == 'orc'
+ else parquet.ParquetWriter(temporary, schema))
+ try:
+ written = False
+ for chunk in chunks:
+ table = (pa.Table.from_batches([chunk])
+ if isinstance(chunk, pa.RecordBatch) else chunk)
+ converted = convert_table(table)
+ if file_format == 'orc':
+ writer.write(converted)
+ else:
+ writer.write_table(converted)
+ written = True
+ if not written:
+ empty = pa.Table.from_batches([], schema=schema)
+ if file_format == 'orc':
+ writer.write(empty)
+ else:
+ writer.write_table(empty)
+ finally:
+ writer.close()
+ temporary.flush()
+ os.fsync(temporary.fileno())
+ os.link(temporary_path, destination)
+ finally:
+ temporary_path.unlink(missing_ok=True)
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('--input', required=True, type=Path)
+ parser.add_argument('--output', required=True, type=Path)
+ parser.add_argument('--format', required=True, choices=('orc', 'parquet'))
+ parser.add_argument('--confirm-legacy-doris-date-encoding', action='store_true', required=True,
+ help='Confirm every DATE field in this file uses the old Doris encoding')
+ args = parser.parse_args()
+ migrate(args.input, args.output, args.format, args.confirm_legacy_doris_date_encoding)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/tools/legacy_date_migration/test_migrate_legacy_dates.py b/tools/legacy_date_migration/test_migrate_legacy_dates.py
new file mode 100644
index 00000000000..69a5d5c1651
--- /dev/null
+++ b/tools/legacy_date_migration/test_migrate_legacy_dates.py
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import pyarrow as pa
+import pyarrow.orc as orc
+import pyarrow.parquet as parquet
+
+from migrate_legacy_dates import convert_array, convert_table, migrate
+
+
+class LegacyDateMigrationTest(unittest.TestCase):
+ def test_boundary_ordinals_and_nulls(self):
+ raw = [-719527, -719526, -719469, -719468, -719162, -1, 0, 19723, 2932896, None]
+ expected = [-719528, -719527, -719470] + raw[3:]
+ result = convert_array(pa.array(raw, pa.date32()))
+ self.assertEqual(expected, result.cast(pa.int32()).to_pylist())
+
+ def test_nested_values_and_slices(self):
+ date = pa.date32()
+ types_and_values = [
+ (pa.list_(date), [[-719527], None, [-719469, None], []]),
+ (pa.large_list(date), [[-719527], None, [-719469, None], []]),
+ (pa.list_(date, 2), [[-719527, 0], None, [-719469, None], [0, 1]]),
+ (pa.struct([('d', date), ('s', pa.string())]),
+ [{'d': -719527, 's': 'a'}, None, {'d': -719469, 's': 'b'}, {'d': None}]),
+ (pa.map_(pa.string(), pa.list_(date)),
+ [[('a', [-719527])], None, [('b', [-719469, None])], []]),
+ (pa.map_(date, pa.string()), [[(-719527, 'a')], None, [(-719469, 'b')], []]),
+ ]
+ for data_type, values in types_and_values:
+ with self.subTest(data_type=data_type):
+ original = pa.array(values, data_type)
+ converted = convert_array(original)
+ self.assertEqual(original.type, converted.type)
+ self.assertEqual(original.is_null(), converted.is_null())
+ self.assertEqual(converted.slice(1, 2), convert_array(original.slice(1, 2)))
+ array = convert_array(pa.array([[-719527, -719469, None]], pa.list_(date)))
+ self.assertEqual([-719528, -719470, None], array.values.cast(pa.int32()).to_pylist())
+
+ def test_null_parent_hides_invalid_date_payload(self):
+ hidden = pa.array([2147483647, -719527], pa.date32())
+ mask = pa.array([True, False])
+ arrays = [
+ pa.StructArray.from_arrays([hidden], names=['d'], mask=mask),
+ pa.ListArray.from_arrays([0, 1, 2], hidden, mask=mask),
+ pa.FixedSizeListArray.from_arrays(hidden, 1, mask=mask),
+ pa.MapArray.from_arrays([0, 1, 2], pa.array(['a', 'b']), hidden, mask=mask),
+ pa.MapArray.from_arrays([0, 1, 2], hidden, pa.array(['a', 'b']), mask=mask),
+ ]
+ for array in arrays:
+ with self.subTest(data_type=array.type):
+ result = convert_array(array)
+ self.assertTrue(result[0].as_py() is None)
+ # The visible second row is identical to converting it independently.
+ self.assertEqual(result.slice(1), convert_array(array.slice(1)))
+
+ def test_unused_dictionary_values_are_not_validated(self):
+ dictionary = pa.array([-719527] + [2147483647] * 199, pa.date32())
+ array = pa.DictionaryArray.from_arrays(pa.array([0, None], pa.int8()), dictionary)
+ result = convert_array(array)
+ self.assertEqual(array.indices, result.indices)
+ self.assertEqual([-719528, None], result.dictionary_decode().cast(pa.int32()).to_pylist())
+ invalid = pa.DictionaryArray.from_arrays(pa.array([1], pa.int8()), dictionary)
+ with self.assertRaises(ValueError):
+ convert_array(invalid)
+
+ def test_metadata_chunks_and_other_types(self):
+ schema = pa.schema([pa.field('d', pa.date32(), metadata={b'field': b'value'}),
+ pa.field('i', pa.int64(), nullable=False)], metadata={b'table': b'value'})
+ table = pa.Table.from_arrays([
+ pa.chunked_array([pa.array([-719527], pa.date32()), pa.array([None], pa.date32())]),
+ pa.array([1, 2], pa.int64())], schema=schema)
+ result = convert_table(table)
+ self.assertTrue(result.schema.equals(schema, check_metadata=True))
+ self.assertEqual(table['i'], result['i'])
+ self.assertEqual(2, result['d'].num_chunks)
+ self.assertEqual([-719528, None], result['d'].cast(pa.int32()).to_pylist())
+
+ def test_round_trip_files_and_no_overwrite(self):
+ with tempfile.TemporaryDirectory() as directory:
+ for fmt in ['orc', 'parquet']:
+ source = Path(directory) / ('source.' + fmt)
+ output = Path(directory) / ('output.' + fmt)
+ table = pa.table({'d': pa.array([-719527, -719469, -719468, 0, 2932896, None],
+ pa.date32())})
+ module = orc if fmt == 'orc' else parquet
+ module.write_table(table, source)
+ before = source.read_bytes()
+ with self.assertRaises(ValueError):
+ migrate(source, output, fmt)
+ migrate(source, output, fmt, True)
+ self.assertEqual([-719528, -719470, -719468, 0, 2932896, None],
+ module.read_table(output)['d'].cast(pa.int32()).to_pylist())
+ with self.assertRaises(FileExistsError):
+ migrate(source, output, fmt, True)
+ with self.assertRaises(ValueError):
+ migrate(source, source, fmt, True)
+ # A corrected lower-bound value is outside the legacy encoding's domain.
+ with self.assertRaises(ValueError):
+ migrate(output, Path(directory) / ('twice.' + fmt), fmt, True)
+ self.assertEqual(before, source.read_bytes())
+ self.assertFalse(list(Path(directory).glob('.legacy-date-*')))
+
+ def test_failure_never_publishes_output(self):
+ with tempfile.TemporaryDirectory() as directory:
+ source, output = Path(directory) / 'source.parquet', Path(directory) / 'output.parquet'
+ parquet.write_table(pa.table({'d': pa.array([-719528], pa.date32())}), source)
+ with self.assertRaises(ValueError):
+ migrate(source, output, 'parquet', True)
+ self.assertFalse(output.exists())
+ self.assertFalse(list(Path(directory).glob('.legacy-date-*')))
+
+ def test_concurrent_output_is_preserved(self):
+ with tempfile.TemporaryDirectory() as directory:
+ source, output = Path(directory) / 'source.parquet', Path(directory) / 'output.parquet'
+ parquet.write_table(pa.table({'d': pa.array([0], pa.date32())}), source)
+ def racing_writer(*args):
+ output.write_bytes(b'other writer')
+ raise FileExistsError(output)
+ with patch('migrate_legacy_dates.os.link', side_effect=racing_writer):
+ with self.assertRaises(FileExistsError):
+ migrate(source, output, 'parquet', True)
+ self.assertEqual(b'other writer', output.read_bytes())
+ self.assertFalse(list(Path(directory).glob('.legacy-date-*')))
+
+ def test_nested_file_values_and_timestamp_precision(self):
+ schema = pa.schema([
+ pa.field('nested', pa.struct([
+ pa.field('dates', pa.list_(pa.date32()), metadata={b'list': b'metadata'}),
+ pa.field('mapping', pa.map_(pa.string(), pa.date32()))])),
+ pa.field('ts', pa.timestamp('us'), nullable=False)], metadata={b'table': b'metadata'})
+ table = pa.Table.from_arrays([
+ pa.array([{'dates': [-719527, None], 'mapping': [('a', -719469)]}, None],
+ schema.field('nested').type),
+ pa.array([123456789, 987654321], pa.timestamp('us'))], schema=schema)
+ expected = convert_table(table)
+ with tempfile.TemporaryDirectory() as directory:
+ for fmt, module in [('orc', orc), ('parquet', parquet)]:
+ source, output = Path(directory) / ('s.' + fmt), Path(directory) / ('o.' + fmt)
+ module.write_table(table, source)
+ migrate(source, output, fmt, True)
+ actual = module.read_table(output)
+ self.assertEqual(expected['nested'].combine_chunks(),
+ actual['nested'].combine_chunks().cast(schema.field('nested').type))
+ self.assertEqual(expected['ts'], actual['ts'].cast(pa.timestamp('us')))
+ if fmt == 'parquet':
+ self.assertTrue(actual.schema.equals(module.read_table(source).schema,
+ check_metadata=True))
+
+ def test_empty_files(self):
+ with tempfile.TemporaryDirectory() as directory:
+ for fmt, module in [('orc', orc), ('parquet', parquet)]:
+ source, output = Path(directory) / ('s.' + fmt), Path(directory) / ('o.' + fmt)
+ module.write_table(pa.table({'d': pa.array([], pa.date32())}), source)
+ migrate(source, output, fmt, True)
+ self.assertEqual(pa.date32(), module.read_table(output).schema.field('d').type)
+
+
+if __name__ == '__main__':
+ unittest.main()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org