You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2026/10/20 01:28:31 UTC
(doris) branch branch-4.2 updated: [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon (branch-4.2) (#69163)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 079797b1b68 [feat](datalake) Support nanosecond timestamps in Iceberg and Paimon (branch-4.2) (#69163)
079797b1b68 is described below
commit 079797b1b68b82aefe469437cfca668b34e2d00a
Author: Gabriel <li...@selectdb.com>
AuthorDate: Tue Oct 20 09:28:22 2026 +0800
[feat](datalake) Support nanosecond timestamps in Iceberg and Paimon (branch-4.2) (#69163)
### What problem does this PR solve?
Backport #68934 to branch-4.2, preserving nanosecond timestamps through
Iceberg and Paimon reads and writes, including nested values. Align
external UUID mappings with the native UUID type already available on
branch-4.2.
Only changed mappings are listed below; ordinary external text remains
STRING.
| Changed external type | Previous Doris mapping | New Doris mapping |
| --- | --- | --- |
| Iceberg `timestamp_ns` without time zone | Unsupported |
`TIMESTAMP_NS` |
| Paimon `timestamp(7)`, `timestamp(8)`, `timestamp(9)` |
`DATETIMEV2(6)` | `TIMESTAMP_NS` |
| Unannotated ORC `BINARY`, including nested fields, in V2 | `STRING` by
default | `VARBINARY` |
| Unannotated Parquet `BYTE_ARRAY` / `FIXED_LEN_BYTE_ARRAY`, in V2 |
`STRING` by default | `VARBINARY` |
| Parquet UUID logical annotation, in V2 | UUID, STRING or VARBINARY
depending on reader context/options | `UUID` |
| ORC UUID logical annotation, in V2 | UUID or VARBINARY depending on
options; Iceberg UUID used a binary carrier | `UUID` |
| Iceberg `uuid` | `VARBINARY(16)` | `UUID` |
| PostgreSQL `uuid` and SQL Server `uniqueidentifier` through JDBC |
`STRING` | `UUID` |
| Trino `uuid` through JDBC or the Trino connector | Unsupported |
`UUID` |
| Hudi Avro STRING with UUID logical annotation | `STRING` | `UUID` |
These mappings also apply recursively to supported complex types. Binary
mapping options no longer override UUID annotations. UUID transport,
defaults, equality deletes, identity/bucket partition writes and
supported JDBC bindings preserve native UUID values. Iceberg UUID text
inputs are validated before ordinary sink coercion, including VALUES,
nested collections and dynamic expressions.
Doris `TIMESTAMP_NS` writes as Iceberg `timestamp_ns` or Paimon
`timestamp(9)`. Iceberg nanosecond schemas require format version 3 and
FileScanner V2. Paimon high-precision reads use JNI. Precision 7/8
writes reject narrowing that would leave the signed-nanosecond range.
Iceberg planning keeps lossy casts as residual predicates, normalizes
raw rewrite literals using the session timezone, and retains equality
deletes when historical DATE bounds cannot safely be compared with nanos
bounds. Static partition parsing accepts the same compact timestamp/UUID
forms as the sink. ORC reads stripe timezones lazily and amortizes
RowReader setup; COUNT of nanos-bearing values falls back to decoding
for range validation, while COUNT(*) remains metadata-only.
Scanner V1 and backward compatibility changes are outside this update's
scope. Existing non-UUID V1 regression coverage is retained. Branch-4.2
internal tables still reject VARBINARY; the regression verifies that
existing CTAS limitation explicitly.
### Release note
Support Iceberg/Paimon nanosecond timestamps and native external UUID
mappings on branch-4.2; infer external binary types independently of the
legacy mapping toggle.
### Validation
- Full FE and BE builds; FE Checkstyle and clang-format 16 for every
affected C/C++ file.
- 413 focused BE tests under ASAN, plus focused FE/JNI/Paimon/Trino unit
tests covering the review fixes and UUID mappings.
- Local regression coverage includes UUID export/readback, PostgreSQL
JDBC and CDC snapshot/incremental reads, HDFS Parquet group 6, Iceberg
nanos/UUID/defaults/write inputs, and Paimon variant shredding. Changed
golden files are generated by execution and verified in comparison mode.
- The CI FE performance-threshold failure passes locally with all 36
tests in its class. The coverage failure occurs during Maven bootstrap
download, before tests. Both are rerun by the PR build trigger.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Regression test
- Behavior changed:
- [x] Yes, as listed in the mapping table.
- Does this need documentation?
- [x] Yes, external type mapping documentation should include
TIMESTAMP_NS and native UUID mappings.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
---
.../data_type_serde/data_type_datetimev2_serde.cpp | 16 +-
be/src/core/data_type_serde/data_type_serde.h | 4 +-
.../data_type_timestamp_ns_serde.cpp | 185 +++++-
.../data_type_serde/data_type_timestamp_ns_serde.h | 9 +
.../data_type_serde/data_type_varbinary_serde.cpp | 39 +-
.../data_type_serde/data_type_varbinary_serde.h | 4 +-
be/src/core/data_type_serde/orc_serde_utils.cpp | 18 +-
be/src/core/data_type_serde/orc_serde_utils.h | 3 +
.../orc_timestamp_decode_context.cpp | 38 ++
.../data_type_serde/orc_timestamp_decode_context.h | 44 ++
be/src/core/data_type_serde/parquet_timestamp.h | 31 +
be/src/exec/connector/jni_connector.cpp | 6 +
be/src/exec/connector/jni_connector.h | 4 +-
.../sink/writer/iceberg/partition_transformers.cpp | 21 +
.../sink/writer/iceberg/partition_transformers.h | 21 +-
.../writer/iceberg/viceberg_partition_writer.cpp | 38 +-
.../sink/writer/iceberg/viceberg_table_writer.cpp | 5 +
be/src/format/CMakeLists.txt | 5 +
be/src/format/table/iceberg/arrow_schema_util.cpp | 19 +-
be/src/format/table/iceberg/types.cpp | 8 +-
be/src/format/table/iceberg/types.h | 34 +-
be/src/format/table/iceberg_default_value.h | 2 +-
be/src/format/transformer/vorc_transformer.cpp | 31 +-
be/src/format_v2/column_mapper.cpp | 7 +-
be/src/format_v2/expr/cast.h | 2 +-
be/src/format_v2/expr/evolution_cast.cpp | 41 ++
be/src/format_v2/expr/evolution_cast.h | 40 ++
be/src/format_v2/orc/orc_reader.cpp | 385 ++++++++++--
be/src/format_v2/orc/orc_reader.h | 6 +-
be/src/format_v2/parquet/native_schema_desc.cpp | 22 +-
be/src/format_v2/parquet/native_schema_desc.h | 5 +-
be/src/format_v2/parquet/parquet_reader.cpp | 10 +-
be/src/format_v2/parquet/parquet_reader.h | 3 +-
be/src/format_v2/parquet/parquet_scan.cpp | 1 +
.../parquet/reader/native/column_chunk_reader.cpp | 2 +
be/src/format_v2/schema_reader.cpp | 67 +++
be/src/format_v2/schema_reader.h | 39 ++
be/src/format_v2/table/iceberg_reader.cpp | 15 +-
be/src/format_v2/table_reader.h | 4 +-
be/src/service/internal_service.cpp | 22 +-
.../core/data_type/data_type_timestamp_ns_test.cpp | 44 +-
.../data_type_serde_parquet_test.cpp | 53 ++
.../data_type_serde_varbinary_test.cpp | 32 +
.../orc_timestamp_decode_context_test.cpp | 101 ++++
be/test/exec/connector/jni_connector_test.cpp | 57 ++
.../writer/iceberg/partition_transformers_test.cpp | 31 +-
.../writer/iceberg/viceberg_table_writer_test.cpp | 2 +-
.../table/iceberg/arrow_schema_util_test.cpp | 25 +
be/test/format/table/iceberg/type_test.cpp | 31 +-
.../format/transformer/vorc_transformer_test.cpp | 90 +++
.../format/transformer/vparquet_writer_test.cpp | 52 ++
be/test/format_v2/expr/evolution_cast_test.cpp | 182 ++++++
be/test/format_v2/orc/orc_reader_test.cpp | 645 +++++++++++++++++---
.../orc/orc_timestamp_projection_test.cpp | 146 +++++
.../test_data/GenerateJavaTimestampFixture.java | 52 ++
.../orc/test_data/java_timestamp_nanos.orc | Bin 0 -> 425 bytes
be/test/format_v2/parquet/native_decoder_test.cpp | 31 +
be/test/format_v2/parquet/parquet_reader_test.cpp | 105 +++-
be/test/format_v2/parquet/parquet_schema_test.cpp | 32 +-
be/test/format_v2/table/iceberg_reader_test.cpp | 124 +++-
fe/be-java-extensions/hadoop-hudi-scanner/pom.xml | 13 +
.../apache/doris/hudi/HadoopHudiColumnValue.java | 24 +
.../apache/doris/hudi/HadoopHudiJniScanner.java | 12 +-
.../apache/doris/hudi/HudiUuidTransportTest.java | 220 +++++++
.../org/apache/doris/jdbc/BaseJdbcExecutor.java | 12 +-
.../apache/doris/jdbc/PostgreSQLJdbcExecutor.java | 12 +
.../apache/doris/jdbc/SQLServerJdbcExecutor.java | 16 +
.../org/apache/doris/jdbc/TrinoJdbcExecutor.java | 24 +
.../apache/doris/jdbc/JdbcUuidSemanticsTest.java | 73 +++
fe/be-java-extensions/paimon-connector/pom.xml | 26 +
.../doris/paimon/PaimonArrowBatchAdapter.java | 23 +-
.../doris/paimon/PaimonArrowBatchAdapterTest.java | 148 ++++-
.../trinoconnector/TrinoConnectorColumnValue.java | 7 +
.../TrinoTimestampSemanticsTest.java | 14 +
.../org/apache/doris/analysis/UuidLiteral.java | 8 +
.../java/org/apache/doris/catalog/JdbcTable.java | 4 +
.../apache/doris/datasource/hudi/HudiUtils.java | 11 +-
.../datasource/iceberg/DorisTypeToIcebergType.java | 4 +
.../IcebergConflictDetectionFilterUtils.java | 6 +
.../datasource/iceberg/IcebergNereidsUtils.java | 11 +-
.../datasource/iceberg/IcebergTimestampUtils.java | 94 +++
.../datasource/iceberg/IcebergTransaction.java | 24 +-
.../doris/datasource/iceberg/IcebergUtils.java | 117 ++--
.../iceberg/IcebergUuidWriteConverter.java | 228 +++++++
.../iceberg/IcebergWriteSchemaContext.java | 16 +-
.../iceberg/rewrite/RewriteDataFilePlanner.java | 8 +
.../datasource/iceberg/source/IcebergScanNode.java | 11 +-
.../jdbc/client/JdbcPostgreSQLClient.java | 4 +-
.../jdbc/client/JdbcSQLServerClient.java | 4 +-
.../datasource/jdbc/client/JdbcTrinoClient.java | 2 +
.../doris/datasource/jdbc/source/JdbcScanNode.java | 3 +-
.../paimon/DorisToPaimonTypeVisitor.java | 2 +
.../apache/doris/datasource/paimon/PaimonUtil.java | 23 +-
.../datasource/paimon/source/PaimonScanNode.java | 10 +-
.../paimon/source/PaimonValueConverter.java | 5 +
.../TrinoConnectorExternalTable.java | 2 +
.../doris/nereids/rules/analysis/BindSink.java | 9 +-
.../expressions/literal/TimeStampNsLiteral.java | 7 +-
.../plans/commands/info/ColumnDefinition.java | 8 +
.../trees/plans/commands/insert/InsertUtils.java | 24 +-
.../java/org/apache/iceberg/DeleteFileIndex.java | 12 +-
.../org/apache/doris/analysis/UuidLiteralTest.java | 24 +
.../datasource/ExternalTimestampMappingTest.java | 6 +-
.../doris/datasource/hudi/HudiUuidMappingTest.java | 78 +++
.../iceberg/IcebergDDLAndDMLPlanTest.java | 2 +
.../iceberg/IcebergExternalTableTest.java | 4 +
.../iceberg/IcebergMetadataOpsValidationTest.java | 8 +-
.../datasource/iceberg/IcebergTimestampNsTest.java | 247 ++++++++
.../doris/datasource/iceberg/IcebergUtilsTest.java | 16 +-
.../datasource/iceberg/IcebergUuidMappingTest.java | 149 +++++
.../iceberg/IcebergWriteSchemaContextTest.java | 19 +-
.../rewrite/RewriteDataFilePlannerTest.java | 26 +
.../iceberg/source/IcebergScanNodeTest.java | 178 +++++-
.../jdbc/client/JdbcUuidMappingTest.java | 86 +++
.../datasource/jdbc/source/JdbcScanNodeTest.java | 18 +
.../datasource/paimon/PaimonMetadataOpsTest.java | 28 +-
.../datasource/paimon/PaimonTimestampNsTest.java | 76 +++
.../doris/datasource/paimon/PaimonUtilTest.java | 4 +-
.../plans/commands/info/ColumnDefinitionTest.java | 20 +
.../uuid/test_uuid_orc_varbinary_mapping.out | 41 --
.../datatype_p0/uuid/test_uuid_parquet_native.out | 48 +-
.../uuid/test_uuid_parquet_roundtrip.out | 6 +-
.../iceberg/test_iceberg_initial_defaults.out | 128 +---
.../iceberg/test_iceberg_timestamp_ns.out | 383 ++++++++++++
.../test_iceberg_uuid_binary_compatibility.out | 522 ++++++++--------
.../write/test_iceberg_write_uuid_inputs.out | 154 +++++
.../type_test/select/test_pg_all_types_select.out | 3 +-
.../paimon/paimon_timestamp_types.out | 56 +-
.../external_table_p0/paimon/test_paimon_minio.out | 9 +-
.../paimon/test_paimon_table_properties.out | 3 +-
.../paimon/test_paimon_timestamp_ns.out | 659 +++++++++++++++++++++
.../tvf/orc_tvf/test_hdfs_orc_group0_orc_files.out | 4 +-
.../tvf/orc_tvf/test_hdfs_orc_group7_orc_files.out | 4 +-
.../tvf/test_hdfs_parquet_group0.out | 20 +-
.../tvf/test_hdfs_parquet_group2.out | 72 +--
.../tvf/test_hdfs_parquet_group5.out | 6 +-
.../tvf/test_hdfs_parquet_group6.out | 42 +-
.../cdc/test_streaming_postgres_job_all_type.out | 3 +-
.../cdc/test_streaming_postgres_job_uuid.out | 5 +-
.../datatype_p0/uuid/test_uuid_export.groovy | 2 +-
.../uuid/test_uuid_orc_varbinary_mapping.groovy | 18 +-
.../uuid/test_uuid_parquet_native.groovy | 6 +-
.../uuid/test_uuid_parquet_roundtrip.groovy | 1 +
.../iceberg/test_iceberg_initial_defaults.groovy | 49 +-
.../iceberg/test_iceberg_timestamp_ns.groovy | 174 ++++++
.../test_iceberg_uuid_binary_compatibility.groovy | 55 +-
.../write/test_iceberg_write_uuid_inputs.groovy | 232 ++++++++
.../paimon/paimon_timestamp_types.groovy | 57 +-
.../paimon/test_paimon_minio.groovy | 2 +-
.../test_paimon_rust_reader_eq_for_null.groovy | 10 +-
.../paimon/test_paimon_table_properties.groovy | 2 +-
.../paimon/test_paimon_timestamp_ns.groovy | 90 +++
.../test_paimon_write_variant_shredding.groovy | 1 +
.../tvf/orc_format/test_orc_exception_files.groovy | 15 +-
.../tvf/test_hdfs_binary_mapping.groovy | 63 ++
.../tvf/test_hdfs_parquet_group2.groovy | 1 +
.../tvf/test_hdfs_parquet_group5.groovy | 4 +
.../tvf/test_hdfs_parquet_group6.groovy | 7 +-
.../load_p0/tvf/test_tvf_error_column.groovy | 4 +-
159 files changed, 7332 insertions(+), 990 deletions(-)
diff --git a/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp b/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
index 80b862b59c8..ec4d43cbd0d 100644
--- a/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
@@ -32,6 +32,7 @@
#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
#include "core/data_type_serde/parquet_decode_source.h"
#include "core/data_type_serde/parquet_timestamp.h"
#include "core/types.h"
@@ -53,8 +54,7 @@ static const int64_t micro_to_nano_second = 1000;
namespace {
-Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColumnView& orc_view,
- const cctz::time_zone& timezone) {
+Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColumnView& orc_view) {
const auto* orc_batch = dynamic_cast<const ::orc::TimestampVectorBatch*>(orc_view.batch);
if (orc_batch == nullptr) {
return Status::InternalError("Unexpected ORC timestamp batch type {}",
@@ -73,9 +73,10 @@ Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColum
}
auto& value =
reinterpret_cast<DateV2Value<DateTimeV2ValueType>&>(data[old_data_size + row]);
+ const int64_t seconds = orc_batch->data[source_row];
orc_serde_utils::RoundedOrcTimestamp timestamp;
auto status = orc_serde_utils::round_orc_timestamp_to_microseconds(
- orc_batch->data[source_row], orc_batch->nanoseconds[source_row], ×tamp);
+ seconds, orc_batch->nanoseconds[source_row], ×tamp);
if (!status.ok()) {
data.resize(old_data_size);
return status;
@@ -85,8 +86,9 @@ Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColum
// Instant carry precedes timezone conversion; plain TIMESTAMP carry stays in civil time
// so rounding cannot jump across a daylight-saving gap or fold.
status = orc_serde_utils::orc_timestamp_to_datetime(
- is_timestamp_instant ? timestamp.seconds : orc_batch->data[source_row],
- timestamp.microseconds, timezone, !is_timestamp_instant && timestamp.carry, &value);
+ is_timestamp_instant ? timestamp.seconds : seconds, timestamp.microseconds,
+ orc_view.timestamp_context->timezone_for(*orc_view.file_type, seconds),
+ !is_timestamp_instant && timestamp.carry, &value);
if (!status.ok()) {
data.resize(old_data_size);
return status;
@@ -1010,11 +1012,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_orc(IColumn& column,
DORIS_CHECK(view.batch != nullptr);
const auto kind = view.file_type->getKind();
DORIS_CHECK(kind == ::orc::TypeKind::TIMESTAMP || kind == ::orc::TypeKind::TIMESTAMP_INSTANT);
- DORIS_CHECK(view.timezone != nullptr);
+ DORIS_CHECK(view.timestamp_context != nullptr);
if (orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows) == 0) {
return Status::OK();
}
- return decode_timestamp_orc_values(column, view, *view.timezone);
+ return decode_timestamp_orc_values(column, view);
}
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_serde.h b/be/src/core/data_type_serde/data_type_serde.h
index e44bedc1a49..6807db7b8c6 100644
--- a/be/src/core/data_type_serde/data_type_serde.h
+++ b/be/src/core/data_type_serde/data_type_serde.h
@@ -119,13 +119,15 @@ struct FieldInfo {
int precision = 0;
};
+struct OrcTimestampDecodeContext;
+
struct OrcDecodedColumnView {
const orc::Type* file_type = nullptr;
const orc::Type* selected_type = nullptr;
const orc::ColumnVectorBatch* batch = nullptr;
size_t rows = 0;
const std::vector<size_t>* selected_rows = nullptr;
- const cctz::time_zone* timezone = nullptr;
+ const OrcTimestampDecodeContext* timestamp_context = nullptr;
bool enable_mapping_timestamp_tz = false;
};
diff --git a/be/src/core/data_type_serde/data_type_timestamp_ns_serde.cpp b/be/src/core/data_type_serde/data_type_timestamp_ns_serde.cpp
index a7b47247e14..c8e51907836 100644
--- a/be/src/core/data_type_serde/data_type_timestamp_ns_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_timestamp_ns_serde.cpp
@@ -28,6 +28,10 @@
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
#include "core/data_type_serde/arrow_validation.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
+#include "core/data_type_serde/parquet_timestamp.h"
#include "core/value/vdatetime_value.h"
#include "exprs/function/cast/cast_to_timestamp_ns.h"
#include "util/mysql_row_buffer.h"
@@ -37,6 +41,56 @@ namespace doris {
namespace {
+class TimestampNsParquetConsumer final : public ParquetFixedValueConsumer {
+public:
+ TimestampNsParquetConsumer(IColumn& column, const ParquetDecodeContext& context,
+ ParquetMaterializationState* state = nullptr)
+ : _data(assert_cast<ColumnTimeStampNs&>(column).get_data()),
+ _context(context),
+ _state(state) {}
+
+ Status consume(const uint8_t* values, size_t count, size_t width) override {
+ const auto old_size = _data.size();
+ for (size_t i = 0; i < count; ++i) {
+ int64_t nanos;
+ Status status;
+ if (_context.physical_type == ParquetPhysicalType::INT96) {
+ DORIS_CHECK_EQ(width, sizeof(ParquetInt96Timestamp));
+ status = parquet_int96_timestamp_nanos(
+ unaligned_load<ParquetInt96Timestamp>(values + i * width), &nanos);
+ } else if (_context.physical_type == ParquetPhysicalType::INT64) {
+ DORIS_CHECK_EQ(width, sizeof(int64_t));
+ status = parquet_timestamp_nanos(
+ _context.time_unit, unaligned_load<int64_t>(values + i * width), &nanos);
+ } else {
+ return Status::NotSupported("TIMESTAMP_NS expects Parquet INT64 or INT96");
+ }
+ if (!status.ok()) {
+ if (_state != nullptr && _state->mark_conversion_failure(_data.size())) {
+ _data.emplace_back();
+ continue;
+ }
+ _data.resize(old_size);
+ return status;
+ }
+ _data.emplace_back(nanos);
+ }
+ return Status::OK();
+ }
+
+private:
+ ColumnTimeStampNs::Container& _data;
+ const ParquetDecodeContext& _context;
+ ParquetMaterializationState* _state;
+};
+
+class RejectTimestampNsBinaryConsumer final : public ParquetBinaryValueConsumer {
+public:
+ Status consume(const StringRef*, size_t) override {
+ return Status::NotSupported("Binary Parquet values cannot be materialized as TIMESTAMP_NS");
+ }
+};
+
Status get_nanos_per_arrow_timestamp_unit(arrow::TimeUnit::type unit, int64_t* nanos_per_unit) {
switch (unit) {
case arrow::TimeUnit::SECOND:
@@ -332,6 +386,113 @@ Status DataTypeTimeStampNsSerDe::deserialize_column_from_jsonb_vector(
return DataTypeSerDe::deserialize_column_from_jsonb_vector(column_to, column_from, cast_params);
}
+Status DataTypeTimeStampNsSerDe::read_parquet_dictionary(
+ IColumn& column, ParquetDecodeSource& source, const ParquetDecodeContext& context) const {
+ TimestampNsParquetConsumer consumer(column, context);
+ RejectTimestampNsBinaryConsumer binary;
+ return source.decode_dictionary(consumer, binary);
+}
+
+Status DataTypeTimeStampNsSerDe::read_column_from_parquet(
+ IColumn& column, ParquetDecodeSource& source, const ParquetDecodeContext& context,
+ size_t num_values, ParquetMaterializationState& state) const {
+ TimestampNsParquetConsumer consumer(column, context, &state);
+ if (context.encoding != ParquetValueEncoding::DICTIONARY) {
+ return source.decode_fixed_values(num_values, consumer);
+ }
+ if (state.dictionary_generation != source.dictionary_generation()) {
+ state.typed_dictionary = column.clone_empty();
+ auto* null_map = state.begin_dictionary_conversion(source.dictionary_size());
+ TimestampNsParquetConsumer dictionary(*state.typed_dictionary, context, &state);
+ RejectTimestampNsBinaryConsumer binary;
+ auto status = source.decode_dictionary(dictionary, binary);
+ state.end_dictionary_conversion(null_map);
+ RETURN_IF_ERROR(status);
+ DORIS_CHECK_EQ(state.typed_dictionary->size(), source.dictionary_size());
+ state.dictionary_generation = source.dictionary_generation();
+ }
+ return state.materialize_dictionary(column, source, num_values);
+}
+
+Status DataTypeTimeStampNsSerDe::read_column_from_decoded_values(
+ IColumn& column, const DecodedColumnView& view) const {
+ if (view.value_kind != DecodedValueKind::INT64 && view.value_kind != DecodedValueKind::INT96) {
+ return Status::NotSupported("TIMESTAMP_NS expects decoded INT64 or INT96 values");
+ }
+ if (view.values == nullptr && decoded_column_view_has_non_null_value(view)) {
+ return Status::Corruption("Decoded TIMESTAMP_NS value buffer is null");
+ }
+ auto unit = ParquetTimeUnit::MICROS;
+ if (view.time_unit == DecodedTimeUnit::MILLIS) {
+ unit = ParquetTimeUnit::MILLIS;
+ } else if (view.time_unit == DecodedTimeUnit::NANOS) {
+ unit = ParquetTimeUnit::NANOS;
+ }
+ auto& data = assert_cast<ColumnTimeStampNs&>(column).get_data();
+ const auto old_size = data.size();
+ for (int64_t row = 0; row < view.row_count; ++row) {
+ if (decoded_column_view_row_is_null(view, row)) {
+ data.emplace_back();
+ continue;
+ }
+ int64_t nanos;
+ auto status =
+ view.value_kind == DecodedValueKind::INT96
+ ? parquet_int96_timestamp_nanos(
+ unaligned_load<ParquetInt96Timestamp>(
+ view.values + row * sizeof(ParquetInt96Timestamp)),
+ &nanos)
+ : parquet_timestamp_nanos(
+ unit,
+ unaligned_load<int64_t>(view.values + row * sizeof(int64_t)),
+ &nanos);
+ if (!status.ok()) {
+ if (decoded_column_view_can_null_on_conversion_failure(view)) {
+ decoded_column_view_insert_null_on_conversion_failure(column, view, row);
+ continue;
+ }
+ data.resize(old_size);
+ return status;
+ }
+ data.emplace_back(nanos);
+ }
+ return Status::OK();
+}
+
+Status DataTypeTimeStampNsSerDe::read_column_from_orc(IColumn& column,
+ const OrcDecodedColumnView& view) const {
+ const auto& batch = assert_cast<const orc::TimestampVectorBatch&>(*view.batch);
+ auto& data = assert_cast<ColumnTimeStampNs&>(column).get_data();
+ const auto old_size = data.size();
+ const auto rows = orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows);
+ DORIS_CHECK(view.file_type != nullptr);
+ DORIS_CHECK(view.timestamp_context != nullptr);
+ for (size_t row = 0; row < rows; ++row) {
+ const auto source = orc_serde_utils::orc_source_row_at(row, view.selected_rows);
+ if (orc_serde_utils::orc_row_is_null(batch, source)) {
+ data.emplace_back();
+ continue;
+ }
+ const auto nanos = batch.nanoseconds[source];
+ if (nanos < 0 || nanos >= TimeStampNsValue::NANOS_PER_SECOND) {
+ data.resize(old_size);
+ return Status::DataQualityError("Invalid ORC timestamp nanoseconds: {}", nanos);
+ }
+ DateV2Value<DateTimeV2ValueType> civil;
+ const int64_t seconds = batch.data[source];
+ const auto timezone = view.timestamp_context->timezone_for(*view.file_type, seconds);
+ auto status = orc_serde_utils::orc_timestamp_to_datetime(seconds, nanos / 1000, timezone,
+ false, &civil);
+ TimeStampNsValue value;
+ if (!status.ok() || !value.from_datetime(civil, nanos % 1000)) {
+ data.resize(old_size);
+ return Status::DataQualityError("ORC timestamp is outside the TIMESTAMP_NS range");
+ }
+ data.push_back(value);
+ }
+ return Status::OK();
+}
+
Status DataTypeTimeStampNsSerDe::write_column_to_arrow(const IColumn& column,
const NullMap* null_map,
arrow::ArrayBuilder* array_builder,
@@ -425,7 +586,29 @@ Status DataTypeTimeStampNsSerDe::write_column_to_orc(const std::string& timezone
orc::ColumnVectorBatch* orc_col_batch,
int64_t start, int64_t end, Arena& arena,
const FormatOptions& options) const {
- return Status::NotSupported("DataTypeTimeStampNsSerDe::write_column_to_orc");
+ const auto& data = assert_cast<const ColumnTimeStampNs&>(column).get_data();
+ auto& batch = assert_cast<orc::TimestampVectorBatch&>(*orc_col_batch);
+ for (int64_t row = start; row < end; ++row) {
+ if (batch.notNull[row] == 0) {
+ continue;
+ }
+ int64_t seconds;
+ auto civil = data[row].to_datetime();
+ if (!civil.unix_timestamp(&seconds, timezone)) {
+ return Status::DataQualityError("Cannot convert TIMESTAMP_NS to ORC timestamp");
+ }
+ const auto nanos = data[row].nanosecond();
+ // Match the other timestamp writers: ORC-645 aliases these fractions across the epoch.
+ if (seconds == -1 && nanos >= 1000000) {
+ return Status::NotSupported(
+ "ORC cannot represent pre-epoch timestamp fractions in [-0.999, 0) "
+ "seconds without data loss; use Parquet for these values");
+ }
+ batch.data[row] = seconds;
+ batch.nanoseconds[row] = nanos;
+ }
+ batch.numElements = end - start;
+ return Status::OK();
}
void DataTypeTimeStampNsSerDe::write_one_cell_to_binary(const IColumn& src_column,
diff --git a/be/src/core/data_type_serde/data_type_timestamp_ns_serde.h b/be/src/core/data_type_serde/data_type_timestamp_ns_serde.h
index 97eb42cb2e8..dc408066f90 100644
--- a/be/src/core/data_type_serde/data_type_timestamp_ns_serde.h
+++ b/be/src/core/data_type_serde/data_type_timestamp_ns_serde.h
@@ -76,6 +76,15 @@ public:
const ColumnString& column_from,
CastParameters& cast_params) const override;
+ Status read_column_from_parquet(IColumn& column, ParquetDecodeSource& source,
+ const ParquetDecodeContext& context, size_t num_values,
+ ParquetMaterializationState& state) const override;
+ Status read_parquet_dictionary(IColumn& column, ParquetDecodeSource& source,
+ const ParquetDecodeContext& context) const override;
+ Status read_column_from_decoded_values(IColumn& column,
+ const DecodedColumnView& view) const override;
+ Status read_column_from_orc(IColumn& column, const OrcDecodedColumnView& view) const override;
+
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
const cctz::time_zone& ctz) const override;
diff --git a/be/src/core/data_type_serde/data_type_varbinary_serde.cpp b/be/src/core/data_type_serde/data_type_varbinary_serde.cpp
index be3d990ea91..b13bf5a7073 100644
--- a/be/src/core/data_type_serde/data_type_varbinary_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_varbinary_serde.cpp
@@ -317,25 +317,10 @@ Status DataTypeVarbinarySerDe::read_column_from_arrow(IColumn& column,
return Status::OK();
}
-Status DataTypeVarbinarySerDe::write_column_to_orc(const std::string& timezone,
- const IColumn& column, const NullMap* null_map,
- orc::ColumnVectorBatch* orc_col_batch,
- int64_t start, int64_t end, Arena& arena,
- const FormatOptions& options) const {
- auto* cur_batch = dynamic_cast<orc::StringVectorBatch*>(orc_col_batch);
- const auto& varbinary_column_data = assert_cast<const ColumnVarbinary&>(column).get_data();
-
- for (auto row_id = start; row_id < end; row_id++) {
- cur_batch->data[row_id] = const_cast<char*>(varbinary_column_data[row_id].data());
- cur_batch->length[row_id] = varbinary_column_data[row_id].size();
- }
-
- cur_batch->numElements = end - start;
- return Status::OK();
-}
-
Status DataTypeVarbinarySerDe::read_column_from_orc(IColumn& column,
const OrcDecodedColumnView& view) const {
+ // Keep binary payloads intact; selected rows and NULL placeholders must stay aligned.
+ DORIS_CHECK(view.file_type != nullptr);
DORIS_CHECK(view.file_type->getKind() == orc::BINARY);
const auto* batch = dynamic_cast<const orc::StringVectorBatch*>(view.batch);
DORIS_CHECK(batch != nullptr);
@@ -365,11 +350,31 @@ Status DataTypeVarbinarySerDe::read_column_from_orc(IColumn& column,
if (length < 0) {
return Status::Corruption("Invalid negative ORC binary length {}", length);
}
+ if (length > 0 && data == nullptr) {
+ return Status::Corruption("Missing ORC binary payload");
+ }
binary_column.insert_data(length == 0 ? "" : data, cast_set<size_t>(length));
}
return Status::OK();
}
+Status DataTypeVarbinarySerDe::write_column_to_orc(const std::string& timezone,
+ const IColumn& column, const NullMap* null_map,
+ orc::ColumnVectorBatch* orc_col_batch,
+ int64_t start, int64_t end, Arena& arena,
+ const FormatOptions& options) const {
+ auto* cur_batch = dynamic_cast<orc::StringVectorBatch*>(orc_col_batch);
+ const auto& varbinary_column_data = assert_cast<const ColumnVarbinary&>(column).get_data();
+
+ for (auto row_id = start; row_id < end; row_id++) {
+ cur_batch->data[row_id] = const_cast<char*>(varbinary_column_data[row_id].data());
+ cur_batch->length[row_id] = varbinary_column_data[row_id].size();
+ }
+
+ cur_batch->numElements = end - start;
+ return Status::OK();
+}
+
Status DataTypeVarbinarySerDe::serialize_one_cell_to_json(const IColumn& column, int64_t row_num,
BufferWritable& bw,
FormatOptions& options) const {
diff --git a/be/src/core/data_type_serde/data_type_varbinary_serde.h b/be/src/core/data_type_serde/data_type_varbinary_serde.h
index af0da81e80d..76e111e81b0 100644
--- a/be/src/core/data_type_serde/data_type_varbinary_serde.h
+++ b/be/src/core/data_type_serde/data_type_varbinary_serde.h
@@ -107,13 +107,13 @@ public:
int64_t row_idx, bool col_const,
const FormatOptions& options) const override;
+ Status read_column_from_orc(IColumn& column, const OrcDecodedColumnView& view) const override;
+
Status write_column_to_orc(const std::string& timezone, const IColumn& column,
const NullMap* null_map, orc::ColumnVectorBatch* orc_col_batch,
int64_t start, int64_t end, Arena& arena,
const FormatOptions& options) const override;
- Status read_column_from_orc(IColumn& column, const OrcDecodedColumnView& view) const override;
-
void to_string(const IColumn& column, size_t row_num, BufferWritable& bw,
const FormatOptions& options) const override;
diff --git a/be/src/core/data_type_serde/orc_serde_utils.cpp b/be/src/core/data_type_serde/orc_serde_utils.cpp
index e423307122c..f73f708b629 100644
--- a/be/src/core/data_type_serde/orc_serde_utils.cpp
+++ b/be/src/core/data_type_serde/orc_serde_utils.cpp
@@ -20,9 +20,23 @@
#include "common/cast_set.h"
#include "common/check.h"
#include "core/column/column_array.h"
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
namespace doris::orc_serde_utils {
+bool has_nanosecond_timestamp(const ::orc::Type& type) {
+ if (type.getKind() == ::orc::TIMESTAMP && type.hasAttributeKey("iceberg.timestamp-unit") &&
+ type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
+ return true;
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ if (has_nanosecond_timestamp(*type.getSubtype(child))) {
+ return true;
+ }
+ }
+ return false;
+}
+
size_t orc_decode_row_count(size_t rows, const std::vector<size_t>* selected_rows) {
if (selected_rows == nullptr) {
return rows;
@@ -93,7 +107,9 @@ DecodedColumnView make_orc_decoded_view(const OrcDecodedColumnView& orc_view,
DecodedColumnView view;
view.value_kind = value_kind;
view.row_count = cast_set<int64_t>(orc_decode_row_count(orc_view.rows, orc_view.selected_rows));
- view.timezone = orc_view.timezone;
+ if (orc_view.timestamp_context != nullptr) {
+ view.timezone = &orc_view.timestamp_context->session_timezone;
+ }
return view;
}
diff --git a/be/src/core/data_type_serde/orc_serde_utils.h b/be/src/core/data_type_serde/orc_serde_utils.h
index 71c4c17531e..9dcea8613cd 100644
--- a/be/src/core/data_type_serde/orc_serde_utils.h
+++ b/be/src/core/data_type_serde/orc_serde_utils.h
@@ -18,6 +18,7 @@
#pragma once
#include <cstring>
+#include <orc/Type.hh>
#include <orc/Vector.hh>
#include <vector>
@@ -29,6 +30,8 @@
namespace doris {
namespace orc_serde_utils {
+bool has_nanosecond_timestamp(const ::orc::Type& type);
+
size_t orc_decode_row_count(size_t rows, const std::vector<size_t>* selected_rows);
size_t orc_source_row_at(size_t row, const std::vector<size_t>* selected_rows);
bool orc_row_is_null(const ::orc::ColumnVectorBatch& batch, size_t row);
diff --git a/be/src/core/data_type_serde/orc_timestamp_decode_context.cpp b/be/src/core/data_type_serde/orc_timestamp_decode_context.cpp
new file mode 100644
index 00000000000..c42c7b148a3
--- /dev/null
+++ b/be/src/core/data_type_serde/orc_timestamp_decode_context.cpp
@@ -0,0 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
+
+#include <orc/Type.hh>
+
+#include "common/check.h"
+
+namespace doris {
+
+cctz::time_zone OrcTimestampDecodeContext::timezone_for(const orc::Type& file_type,
+ int64_t seconds) const {
+ const auto kind = file_type.getKind();
+ DORIS_CHECK(kind == orc::TIMESTAMP || kind == orc::TIMESTAMP_INSTANT);
+ // Plain timestamps reconstruct the SDK reader's civil value. Instant-to-DATETIME conversion
+ // uses the session zone, which must not follow a plain sibling's stripe writer timezone.
+ if (kind == orc::TIMESTAMP && writer_timezone) {
+ return writer_timezone(seconds);
+ }
+ return kind == orc::TIMESTAMP ? reader_timezone : session_timezone;
+}
+
+} // namespace doris
diff --git a/be/src/core/data_type_serde/orc_timestamp_decode_context.h b/be/src/core/data_type_serde/orc_timestamp_decode_context.h
new file mode 100644
index 00000000000..2a0609cd8ce
--- /dev/null
+++ b/be/src/core/data_type_serde/orc_timestamp_decode_context.h
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cctz/time_zone.h>
+
+#include <functional>
+
+namespace orc {
+class Type;
+}
+
+namespace doris {
+
+// Reader and session zones have different roles even when a batch mixes both timestamp kinds.
+struct OrcTimestampDecodeContext {
+ explicit OrcTimestampDecodeContext(const cctz::time_zone& timezone)
+ : reader_timezone(timezone), session_timezone(timezone) {}
+
+ cctz::time_zone timezone_for(const orc::Type& file_type, int64_t seconds = 0) const;
+
+ // Java carriers follow the SDK writer-zone rules rather than cctz historical LMT.
+ std::function<cctz::time_zone(int64_t)> writer_timezone;
+
+ cctz::time_zone reader_timezone;
+ cctz::time_zone session_timezone;
+};
+
+} // namespace doris
diff --git a/be/src/core/data_type_serde/parquet_timestamp.h b/be/src/core/data_type_serde/parquet_timestamp.h
index ba2aa686ad4..67e0095da7a 100644
--- a/be/src/core/data_type_serde/parquet_timestamp.h
+++ b/be/src/core/data_type_serde/parquet_timestamp.h
@@ -46,6 +46,37 @@ inline Status validate_parquet_timestamp_micros(int64_t timestamp_micros) {
return Status::OK();
}
+inline Status parquet_timestamp_nanos(ParquetTimeUnit unit, int64_t value, int64_t* result) {
+ int64_t factor = 1000;
+ if (unit == ParquetTimeUnit::MILLIS) {
+ factor = 1000000;
+ } else if (unit == ParquetTimeUnit::NANOS) {
+ factor = 1;
+ }
+ if (__builtin_mul_overflow(value, factor, result)) {
+ return Status::DataQualityError("Parquet timestamp is outside the TIMESTAMP_NS range");
+ }
+ return Status::OK();
+}
+
+inline Status parquet_int96_timestamp_nanos(const ParquetInt96Timestamp& value, int64_t* result) {
+ static constexpr int64_t NANOS_IN_DAY = 86400000000000LL;
+ if (value.nanos_of_day < 0 || value.nanos_of_day >= NANOS_IN_DAY) {
+ return Status::DataQualityError("Invalid Parquet INT96 nanos-of-day: {}",
+ value.nanos_of_day);
+ }
+ // Widen before multiplying: the first representable day's midnight is outside Int64.
+ const __int128 nanos =
+ (static_cast<__int128>(value.julian_day) - 2440588) * NANOS_IN_DAY + value.nanos_of_day;
+ if (nanos < std::numeric_limits<int64_t>::min() ||
+ nanos > std::numeric_limits<int64_t>::max()) {
+ return Status::DataQualityError(
+ "Parquet INT96 timestamp is outside the TIMESTAMP_NS range");
+ }
+ *result = static_cast<int64_t>(nanos);
+ return Status::OK();
+}
+
inline Status parquet_timestamp_micros(ParquetTimeUnit unit, int64_t value, int64_t* result) {
if (unit == ParquetTimeUnit::MILLIS) {
// Validate in the source unit before scaling; signed overflow could otherwise turn an
diff --git a/be/src/exec/connector/jni_connector.cpp b/be/src/exec/connector/jni_connector.cpp
index 6fd7af5ee57..0fca0151562 100644
--- a/be/src/exec/connector/jni_connector.cpp
+++ b/be/src/exec/connector/jni_connector.cpp
@@ -50,6 +50,7 @@ class RuntimeProfile;
namespace doris {
+// The legacy Paimon scanner shares Java's signed nanosecond layout with JniDataBridge.
#define FOR_FIXED_LENGTH_TYPES(M) \
M(PrimitiveType::TYPE_TINYINT, ColumnInt8, Int8) \
M(PrimitiveType::TYPE_BOOLEAN, ColumnUInt8, UInt8) \
@@ -67,6 +68,7 @@ namespace doris {
M(PrimitiveType::TYPE_DATEV2, ColumnDateV2, UInt32) \
M(PrimitiveType::TYPE_DATETIME, ColumnDateTime, Int64) \
M(PrimitiveType::TYPE_DATETIMEV2, ColumnDateTimeV2, UInt64) \
+ M(PrimitiveType::TYPE_TIMESTAMP_NS, ColumnTimeStampNs, Int64) \
M(PrimitiveType::TYPE_TIMESTAMPTZ, ColumnTimeStampTz, UInt64) \
M(PrimitiveType::TYPE_IPV4, ColumnIPv4, IPv4) \
M(PrimitiveType::TYPE_IPV6, ColumnIPv6, IPv6) \
@@ -501,6 +503,8 @@ std::string JniConnector::get_jni_type(const DataTypePtr& data_type) {
buffer << "datetimev2(" << type->get_scale() << ")";
return buffer.str();
}
+ case TYPE_TIMESTAMP_NS:
+ return "timestamp_ns";
case TYPE_TIMESTAMPTZ: {
buffer << "timestamptz(" << type->get_scale() << ")";
return buffer.str();
@@ -598,6 +602,8 @@ std::string JniConnector::get_jni_type_with_different_string(const DataTypePtr&
buffer << "datetimev2(" << data_type->get_scale() << ")";
return buffer.str();
}
+ case TYPE_TIMESTAMP_NS:
+ return "timestamp_ns";
case TYPE_TIMESTAMPTZ: {
buffer << "timestamptz(" << data_type->get_scale() << ")";
return buffer.str();
diff --git a/be/src/exec/connector/jni_connector.h b/be/src/exec/connector/jni_connector.h
index d069eb58193..ffa81976f09 100644
--- a/be/src/exec/connector/jni_connector.h
+++ b/be/src/exec/connector/jni_connector.h
@@ -370,6 +370,7 @@ private:
!std::is_same_v<COLUMN_TYPE, ColumnDateTime> &&
!std::is_same_v<COLUMN_TYPE, ColumnDateV2> &&
!std::is_same_v<COLUMN_TYPE, ColumnDateTimeV2> &&
+ !std::is_same_v<COLUMN_TYPE, ColumnTimeStampNs> &&
!std::is_same_v<COLUMN_TYPE, ColumnTimeStampTz>)
static Status _fill_fixed_length_column(MutableColumnPtr& doris_column, CPP_TYPE* ptr,
size_t num_rows) {
@@ -382,7 +383,8 @@ private:
template <typename COLUMN_TYPE, typename CPP_TYPE>
requires(std::is_same_v<COLUMN_TYPE, ColumnDate> ||
- std::is_same_v<COLUMN_TYPE, ColumnDateTime>)
+ std::is_same_v<COLUMN_TYPE, ColumnDateTime> ||
+ std::is_same_v<COLUMN_TYPE, ColumnTimeStampNs>)
static Status _fill_fixed_length_column(MutableColumnPtr& doris_column, CPP_TYPE* ptr,
size_t num_rows) {
auto& column_data = assert_cast<COLUMN_TYPE&>(*doris_column).get_data();
diff --git a/be/src/exec/sink/writer/iceberg/partition_transformers.cpp b/be/src/exec/sink/writer/iceberg/partition_transformers.cpp
index ab6265dcbb2..d1f141792cc 100644
--- a/be/src/exec/sink/writer/iceberg/partition_transformers.cpp
+++ b/be/src/exec/sink/writer/iceberg/partition_transformers.cpp
@@ -115,6 +115,9 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
return std::make_unique<StringBucketPartitionColumnTransform>(source_type,
parsed_width);
}
+ case TYPE_UUID:
+ return std::make_unique<ByteBucketPartitionColumnTransform<ColumnUUID>>(
+ source_type, parsed_width);
case TYPE_VARBINARY: {
return std::make_unique<BinaryBucketPartitionColumnTransform>(source_type,
parsed_width);
@@ -129,6 +132,9 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
case TYPE_DATETIMEV2:
return std::make_unique<TimestampBucketPartitionColumnTransform<TYPE_DATETIMEV2>>(
source_type, parsed_width);
+ case TYPE_TIMESTAMP_NS:
+ return std::make_unique<TimestampBucketPartitionColumnTransform<TYPE_TIMESTAMP_NS>>(
+ source_type, parsed_width);
case TYPE_DECIMALV2: {
return std::make_unique<DecimalBucketPartitionColumnTransform<TYPE_DECIMALV2>>(
source_type, parsed_width);
@@ -171,6 +177,9 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
case TYPE_DATETIMEV2:
return std::make_unique<TimestampYearPartitionColumnTransform<TYPE_DATETIMEV2>>(
source_type);
+ case TYPE_TIMESTAMP_NS:
+ return std::make_unique<TimestampYearPartitionColumnTransform<TYPE_TIMESTAMP_NS>>(
+ source_type);
default: {
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
"Unsupported type {} for partition column transform {}",
@@ -188,6 +197,9 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
case TYPE_DATETIMEV2:
return std::make_unique<TimestampMonthPartitionColumnTransform<TYPE_DATETIMEV2>>(
source_type);
+ case TYPE_TIMESTAMP_NS:
+ return std::make_unique<TimestampMonthPartitionColumnTransform<TYPE_TIMESTAMP_NS>>(
+ source_type);
default: {
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
"Unsupported type {} for partition column transform {}",
@@ -205,6 +217,9 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
case TYPE_DATETIMEV2:
return std::make_unique<TimestampDayPartitionColumnTransform<TYPE_DATETIMEV2>>(
source_type);
+ case TYPE_TIMESTAMP_NS:
+ return std::make_unique<TimestampDayPartitionColumnTransform<TYPE_TIMESTAMP_NS>>(
+ source_type);
default: {
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
"Unsupported type {} for partition column transform {}",
@@ -219,6 +234,9 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
case TYPE_DATETIMEV2:
return std::make_unique<TimestampHourPartitionColumnTransform<TYPE_DATETIMEV2>>(
source_type);
+ case TYPE_TIMESTAMP_NS:
+ return std::make_unique<TimestampHourPartitionColumnTransform<TYPE_TIMESTAMP_NS>>(
+ source_type);
default: {
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
"Unsupported type {} for partition column transform {}",
@@ -268,6 +286,7 @@ std::string PartitionColumnTransform::get_partition_value(const DataTypePtr type
case TYPE_DOUBLE: {
return floating_point_partition_value_to_string(std::any_cast<Float64>(value));
}
+ case TYPE_UUID:
case TYPE_VARCHAR:
case TYPE_CHAR:
case TYPE_STRING: {
@@ -303,6 +322,8 @@ std::string PartitionColumnTransform::get_partition_value(const DataTypePtr type
return std::any_cast<TimestampTzValue>(value).to_string(cctz::utc_time_zone(),
type->get_scale());
}
+ case TYPE_TIMESTAMP_NS:
+ return std::any_cast<TimeStampNsValue>(value).to_string();
case TYPE_DATETIMEV2: {
char buf[64];
char* pos = std::any_cast<DateV2Value<DateTimeV2ValueType>>(value).to_string(
diff --git a/be/src/exec/sink/writer/iceberg/partition_transformers.h b/be/src/exec/sink/writer/iceberg/partition_transformers.h
index da63554ff03..d6b397d4beb 100644
--- a/be/src/exec/sink/writer/iceberg/partition_transformers.h
+++ b/be/src/exec/sink/writer/iceberg/partition_transformers.h
@@ -19,6 +19,8 @@
#include "core/column/column_varbinary.h"
#include "core/data_type/data_type_factory.hpp"
+#include "core/value/timestamp_ns_value.h"
+#include "core/value/uuid_value.h"
#include "exec/common/stringop_substring.h"
#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
#include "exprs/function/cast/cast_to_datev2_impl.hpp"
@@ -75,6 +77,12 @@ public:
return value.utc_dt();
}
+ static DateV2Value<DateTimeV2ValueType> timestamp_value(const TimeStampNsValue& value) {
+ // Iceberg bucket hashing uses epoch microseconds even for timestamp_ns; calendar
+ // transforms also ignore sub-microsecond digits. to_datetime floors negative fractions.
+ return value.to_datetime();
+ }
+
static std::string human_year(int year_ordinal) {
auto ymd = std::chrono::year_month_day {EPOCH} + std::chrono::years(year_ordinal);
return std::to_string(static_cast<int>(ymd.year()));
@@ -775,6 +783,8 @@ public:
std::string name() const override {
if constexpr (std::is_same_v<ColumnType, ColumnVarbinary>) {
return "BinaryBucket";
+ } else if constexpr (std::is_same_v<ColumnType, ColumnUUID>) {
+ return "UuidBucket";
}
return "StringBucket";
}
@@ -806,8 +816,15 @@ public:
out_data.resize(row_count);
for (size_t row = 0; row < row_count; ++row) {
// Iceberg hashes raw bytes for both strings and binary, without text decoding.
- const auto bytes = str_col->get_data_at(row);
- uint32_t hash_value = HashUtil::murmur_hash3_32(bytes.data, bytes.size, 0);
+ uint32_t hash_value;
+ if constexpr (std::is_same_v<ColumnType, ColumnUUID>) {
+ // Iceberg hashes network-order UUID bytes, never the native integer layout.
+ const auto bytes = UUIDValue::to_big_endian(str_col->get_data()[row]);
+ hash_value = HashUtil::murmur_hash3_32(bytes.data(), bytes.size(), 0);
+ } else {
+ const auto bytes = str_col->get_data_at(row);
+ hash_value = HashUtil::murmur_hash3_32(bytes.data, bytes.size, 0);
+ }
out_data[row] = (hash_value & INT32_MAX) % _bucket_num;
}
diff --git a/be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp b/be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp
index e1d2de9c439..565e47b4686 100644
--- a/be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp
+++ b/be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp
@@ -29,6 +29,32 @@
#include "runtime/runtime_state.h"
namespace doris {
+namespace {
+bool contains_nanosecond_timestamp(const iceberg::Type& type) {
+ switch (type.type_id()) {
+ case iceberg::TypeID::TIMESTAMP_NS:
+ case iceberg::TypeID::TIMESTAMPTZ_NS:
+ return true;
+ case iceberg::TypeID::STRUCT:
+ for (const auto& field : static_cast<const iceberg::StructType&>(type).fields()) {
+ if (contains_nanosecond_timestamp(*field.field_type())) {
+ return true;
+ }
+ }
+ return false;
+ case iceberg::TypeID::LIST:
+ return contains_nanosecond_timestamp(
+ *static_cast<const iceberg::ListType&>(type).element_field().field_type());
+ case iceberg::TypeID::MAP: {
+ const auto& map = static_cast<const iceberg::MapType&>(type);
+ return contains_nanosecond_timestamp(*map.key_field().field_type()) ||
+ contains_nanosecond_timestamp(*map.value_field().field_type());
+ }
+ default:
+ return false;
+ }
+}
+} // namespace
VIcebergPartitionWriter::VIcebergPartitionWriter(
const TDataSink& t_sink, std::vector<std::string> partition_values,
@@ -96,10 +122,14 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil
to_string(_compress_type));
}
}
- ParquetFileOptions parquet_options = {.compression_type = parquet_compression_type,
- .parquet_version = TParquetVersion::PARQUET_1_0,
- .parquet_disable_dictionary = false,
- .enable_int96_timestamps = false};
+ // Parquet 1.0 coerces Arrow nanoseconds to microseconds, including nested fields.
+ ParquetFileOptions parquet_options = {
+ .compression_type = parquet_compression_type,
+ .parquet_version = contains_nanosecond_timestamp(_schema.root_struct())
+ ? TParquetVersion::PARQUET_2_LATEST
+ : TParquetVersion::PARQUET_1_0,
+ .parquet_disable_dictionary = false,
+ .enable_int96_timestamps = false};
_file_format_transformer = std::make_unique<VIcebergParquetWriter>(
state, _file_writer.get(), _write_output_expr_ctxs, _write_column_names, false,
parquet_options, _iceberg_schema_json, _schema);
diff --git a/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp b/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp
index 71c5c1c931d..23d7e9b30a0 100644
--- a/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp
+++ b/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp
@@ -973,6 +973,9 @@ std::any VIcebergTableWriter::_get_iceberg_partition_value(
case TYPE_DOUBLE: {
return *reinterpret_cast<const Float64*>(item);
}
+ case TYPE_UUID:
+ // Identity partition paths and commit metadata use canonical UUID text.
+ return UUIDValue::to_string(assert_cast<const ColumnUUID&>(*col_ptr).get_data()[position]);
case TYPE_VARBINARY:
case TYPE_VARCHAR:
case TYPE_CHAR:
@@ -991,6 +994,8 @@ std::any VIcebergTableWriter::_get_iceberg_partition_value(
case TYPE_DATETIMEV2: {
return binary_cast<uint64_t, DateV2Value<DateTimeV2ValueType>>(*(int64_t*)item);
}
+ case TYPE_TIMESTAMP_NS:
+ return assert_cast<const ColumnTimeStampNs&>(*col_ptr).get_data()[position];
case TYPE_TIMESTAMPTZ: {
// Keep the UTC carrier typed through identity routing and FE commit serialization.
return assert_cast<const ColumnTimeStampTz&>(*col_ptr).get_data()[position];
diff --git a/be/src/format/CMakeLists.txt b/be/src/format/CMakeLists.txt
index f73ef2dbfaa..4cf1b1abd70 100644
--- a/be/src/format/CMakeLists.txt
+++ b/be/src/format/CMakeLists.txt
@@ -28,4 +28,9 @@ list(APPEND SRC_FILES ${FORMAT_V2_SRC_FILES})
add_library(Format STATIC ${SRC_FILES})
+# Read the pinned SDK timezone rules without changing SDK decoding or public interfaces.
+target_include_directories(Format PRIVATE
+ ${CMAKE_CURRENT_SOURCE_DIR}/../../../contrib/apache-orc/c++/src
+ ${BUILD_DIR}/apache-orc/c++/src)
+
pch_reuse(Format)
diff --git a/be/src/format/table/iceberg/arrow_schema_util.cpp b/be/src/format/table/iceberg/arrow_schema_util.cpp
index f82aa27b3cc..2380f657cc7 100644
--- a/be/src/format/table/iceberg/arrow_schema_util.cpp
+++ b/be/src/format/table/iceberg/arrow_schema_util.cpp
@@ -84,12 +84,21 @@ Status IcebergArrowSchemaConvertor::convert_to_arrow_field(
arrow_type = arrow::date32();
break;
- case iceberg::TypeID::TIMESTAMP: {
- iceberg::TimestampType* t_type = static_cast<iceberg::TimestampType*>(field.field_type());
- std::string real_tz = t_type->should_adjust_to_utc() ? _timezone : "";
- arrow_type = std::make_shared<arrow::TimestampType>(arrow::TimeUnit::MICRO, real_tz);
+ case iceberg::TypeID::TIMESTAMP:
+ arrow_type = arrow::timestamp(arrow::TimeUnit::MICRO);
+ break;
+
+ case iceberg::TypeID::TIMESTAMPTZ:
+ arrow_type = arrow::timestamp(arrow::TimeUnit::MICRO, _timezone);
+ break;
+
+ case iceberg::TypeID::TIMESTAMP_NS:
+ arrow_type = arrow::timestamp(arrow::TimeUnit::NANO);
+ break;
+
+ case iceberg::TypeID::TIMESTAMPTZ_NS:
+ arrow_type = arrow::timestamp(arrow::TimeUnit::NANO, _timezone);
break;
- }
case iceberg::TypeID::STRING:
arrow_type = arrow::utf8();
diff --git a/be/src/format/table/iceberg/types.cpp b/be/src/format/table/iceberg/types.cpp
index e9c9edb2900..0f717aa9e87 100644
--- a/be/src/format/table/iceberg/types.cpp
+++ b/be/src/format/table/iceberg/types.cpp
@@ -162,9 +162,13 @@ std::unique_ptr<PrimitiveType> Types::from_primitive_string(const std::string& t
} else if (lower_type_string == "time") {
return std::make_unique<TimeType>();
} else if (lower_type_string == "timestamptz") {
- return std::make_unique<TimestampType>(true);
+ return std::make_unique<TimestampTzType>();
} else if (lower_type_string == "timestamp") {
- return std::make_unique<TimestampType>(false);
+ return std::make_unique<TimestampType>();
+ } else if (lower_type_string == "timestamp_ns") {
+ return std::make_unique<TimestampNsType>();
+ } else if (lower_type_string == "timestamptz_ns") {
+ return std::make_unique<TimestampTzNsType>();
} else if (lower_type_string == "string") {
return std::make_unique<StringType>();
} else if (lower_type_string == "uuid") {
diff --git a/be/src/format/table/iceberg/types.h b/be/src/format/table/iceberg/types.h
index 6a54146dcd7..85263c79074 100644
--- a/be/src/format/table/iceberg/types.h
+++ b/be/src/format/table/iceberg/types.h
@@ -42,6 +42,9 @@ enum TypeID {
DATE,
TIME,
TIMESTAMP,
+ TIMESTAMPTZ,
+ TIMESTAMP_NS,
+ TIMESTAMPTZ_NS,
STRING,
UUID,
FIXED,
@@ -324,26 +327,29 @@ public:
std::string to_string() const override { return "string"; }
};
+// Keep unit and timezone semantics in the type identity so dispatch cannot silently lose them.
class TimestampType : public PrimitiveType {
public:
- ~TimestampType() override = default;
-
- TimestampType(bool adjust_to_utc) : _adjust_to_utc(adjust_to_utc) {}
-
TypeID type_id() const override { return TypeID::TIMESTAMP; }
+ std::string to_string() const override { return "timestamp"; }
+};
- bool should_adjust_to_utc() const { return _adjust_to_utc; }
+class TimestampTzType : public PrimitiveType {
+public:
+ TypeID type_id() const override { return TypeID::TIMESTAMPTZ; }
+ std::string to_string() const override { return "timestamptz"; }
+};
- std::string to_string() const override {
- if (should_adjust_to_utc()) {
- return "timestamptz";
- } else {
- return "timestamp";
- }
- }
+class TimestampNsType : public PrimitiveType {
+public:
+ TypeID type_id() const override { return TypeID::TIMESTAMP_NS; }
+ std::string to_string() const override { return "timestamp_ns"; }
+};
-private:
- bool _adjust_to_utc;
+class TimestampTzNsType : public PrimitiveType {
+public:
+ TypeID type_id() const override { return TypeID::TIMESTAMPTZ_NS; }
+ std::string to_string() const override { return "timestamptz_ns"; }
};
class TimeType : public PrimitiveType {
diff --git a/be/src/format/table/iceberg_default_value.h b/be/src/format/table/iceberg_default_value.h
index 4845550c58e..4f5ad80d3ce 100644
--- a/be/src/format/table/iceberg_default_value.h
+++ b/be/src/format/table/iceberg_default_value.h
@@ -140,7 +140,7 @@ inline std::string json_scalar_text(const rapidjson::Value& value) {
inline Status normalize_timestamp_for_doris(PrimitiveType primitive_type,
const cctz::time_zone* timezone, std::string* value) {
if (primitive_type != TYPE_DATETIME && primitive_type != TYPE_DATETIMEV2 &&
- primitive_type != TYPE_TIMESTAMPTZ) {
+ primitive_type != TYPE_TIMESTAMPTZ && primitive_type != TYPE_TIMESTAMP_NS) {
return Status::OK();
}
if (const size_t separator = value->find('T'); separator != std::string::npos) {
diff --git a/be/src/format/transformer/vorc_transformer.cpp b/be/src/format/transformer/vorc_transformer.cpp
index ab92ecf0875..3c6da756c85 100644
--- a/be/src/format/transformer/vorc_transformer.cpp
+++ b/be/src/format/transformer/vorc_transformer.cpp
@@ -47,6 +47,7 @@
#include "core/data_type/data_type_map.h"
#include "core/data_type/data_type_struct.h"
#include "core/data_type/define_primitive_type.h"
+#include "core/data_type_serde/orc_serde_utils.h"
#include "core/pod_array.h"
#include "core/string_ref.h"
#include "core/types.h"
@@ -242,6 +243,11 @@ Status VOrcTransformer::open() {
}
}
+ if (orc_serde_utils::has_nanosecond_timestamp(*_schema)) {
+ // ORC and cctz use different ancient local offsets. A UTC wall-clock carrier avoids
+ // shifting nanosecond boundary dates when another session reads the file.
+ _write_options->setTimezoneName("GMT");
+ }
_output_stream = std::make_unique<VOrcOutputStream>(_file_writer);
try {
_write_options->setMemoryPool(ExecEnv::GetInstance()->orc_memory_pool());
@@ -345,6 +351,11 @@ std::unique_ptr<orc::Type> VOrcTransformer::_build_orc_type(
type = orc::createPrimitiveType(orc::DATE);
break;
}
+ case TYPE_TIMESTAMP_NS: {
+ type = orc::createPrimitiveType(orc::TIMESTAMP);
+ type->setAttribute("iceberg.timestamp-unit", "NANOS");
+ break;
+ }
case TYPE_DATETIMEV2: {
type = orc::createPrimitiveType(orc::TIMESTAMP);
break;
@@ -420,6 +431,11 @@ std::unique_ptr<orc::Type> VOrcTransformer::_build_orc_type(
};
switch (nested_field->field_type()->type_id()) {
case iceberg::TypeID::UUID:
+ // Native UUID SerDe already emits network-order bytes and retains its annotation.
+ if (primitive_type == TYPE_UUID) {
+ type->setAttribute(ICEBERG_BINARY_TYPE, "UUID");
+ break;
+ }
use_iceberg_binary_type("UUID");
break;
case iceberg::TypeID::FIXED:
@@ -622,6 +638,10 @@ bool VOrcTransformer::_collect_column_bounds(const orc::ColumnStatistics* col_st
}
} else if (const auto* ts_stats =
dynamic_cast<const orc::TimestampColumnStatistics*>(col_stats)) {
+ if (remove_nullable(data_type)->get_primitive_type() == TYPE_TIMESTAMP_NS) {
+ // Omit bounds rather than publish microseconds as Iceberg nanoseconds.
+ return false;
+ }
if (ts_stats->hasMinimum() && ts_stats->hasMaximum()) {
// ORC stores milliseconds plus a sub-millisecond nanos tail. Dropping that tail
// makes Iceberg's upper bound smaller than real rows and causes false-negative pruning.
@@ -723,9 +743,9 @@ Status VOrcTransformer::write(const Block& block) {
normalized_columns.push_back(std::move(raw_column));
const auto& write_column = normalized_columns.back();
RETURN_IF_ERROR(_resize_row_batch(col.type, *write_column, root->fields[i]));
- RETURN_IF_ERROR(_serdes[i]->write_column_to_orc(_state->timezone(), *write_column,
- nullptr, root->fields[i], 0, sz, arena,
- options));
+ RETURN_IF_ERROR(_serdes[i]->write_column_to_orc(_write_options->getTimezoneName(),
+ *write_column, nullptr, root->fields[i],
+ 0, sz, arena, options));
}
root->numElements = sz;
_writer->add(*row_batch);
@@ -923,6 +943,11 @@ static Status normalize_iceberg_binary_column(const ColumnPtr& column, const Dat
switch (nested_field.field_type()->type_id()) {
case iceberg::TypeID::UUID:
+ if (type->get_primitive_type() == TYPE_UUID) {
+ // Reinterpreting the native integer layout here would swap UUID bytes.
+ *normalized_column = column;
+ return Status::OK();
+ }
return normalize_iceberg_uuid_column(column, normalized_column, skipped_rows);
case iceberg::TypeID::FIXED:
return normalize_iceberg_fixed_column(column, type, nested_field, normalized_column,
diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp
index b5f7e7c89bf..5839ececd7c 100644
--- a/be/src/format_v2/column_mapper.cpp
+++ b/be/src/format_v2/column_mapper.cpp
@@ -51,6 +51,7 @@
#include "exprs/vslot_ref.h"
#include "format_v2/column_mapper_nested.h"
#include "format_v2/expr/cast.h"
+#include "format_v2/expr/evolution_cast.h"
#include "format_v2/file_reader.h"
#include "format_v2/schema_projection.h"
#include "format_v2/table_reader.h"
@@ -1096,7 +1097,9 @@ static VExprSPtr cast_file_expr_to_table_type(const VExprSPtr& file_expr,
DORIS_CHECK(file_expr != nullptr);
DORIS_CHECK(table_type != nullptr);
DORIS_CHECK(rewrite_context != nullptr);
- auto cast_expr = Cast::create_shared(table_type);
+ // Localized filters perform the same evolution as projection; SQL non-strict casts must
+ // not replace an overflowing retained DATE with NULL and filter it out before materialization.
+ auto cast_expr = EvolutionCast::create_shared(table_type);
cast_expr->add_child(file_expr);
rewrite_context->add_created_expr(cast_expr);
return cast_expr;
@@ -2168,7 +2171,7 @@ static void rebuild_projection(ColumnMapping* mapping, LocalIndex block_position
return;
}
- auto expr = Cast::create_shared(mapping->table_type);
+ auto expr = EvolutionCast::create_shared(mapping->table_type);
expr->add_child(VSlotRef::create_shared(cast_set<int>(block_position.value()),
cast_set<int>(block_position.value()), -1,
mapping->file_type, mapping->file_column_name));
diff --git a/be/src/format_v2/expr/cast.h b/be/src/format_v2/expr/cast.h
index 22604455e50..ed9b0742812 100644
--- a/be/src/format_v2/expr/cast.h
+++ b/be/src/format_v2/expr/cast.h
@@ -34,7 +34,7 @@ class VExprContext;
namespace doris::format {
-class Cast final : public VExpr {
+class Cast : public VExpr {
ENABLE_FACTORY_CREATOR(Cast);
public:
diff --git a/be/src/format_v2/expr/evolution_cast.cpp b/be/src/format_v2/expr/evolution_cast.cpp
new file mode 100644
index 00000000000..70d878474f7
--- /dev/null
+++ b/be/src/format_v2/expr/evolution_cast.cpp
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/expr/evolution_cast.h"
+
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/vexpr_context.h"
+
+namespace doris::format {
+
+Status EvolutionCast::prepare(RuntimeState* state, const RowDescriptor& desc,
+ VExprContext* context) {
+ RETURN_IF_ERROR(Cast::prepare(state, desc, context));
+ if (remove_nullable(_children[0]->data_type())->get_primitive_type() == TYPE_DATEV2 &&
+ remove_nullable(_data_type)->get_primitive_type() == TYPE_TIMESTAMP_NS) {
+ // Iceberg DATE promotion must fail outside the nanosecond range, even in non-strict SQL
+ // sessions. Otherwise a localized IS NOT NULL filter can silently discard retained rows.
+ context->fn_context(_fn_context_index)->set_enable_strict_mode(true);
+ }
+ return Status::OK();
+}
+
+std::string EvolutionCast::debug_string() const {
+ return "Evolution" + Cast::debug_string();
+}
+
+} // namespace doris::format
diff --git a/be/src/format_v2/expr/evolution_cast.h b/be/src/format_v2/expr/evolution_cast.h
new file mode 100644
index 00000000000..25b5c02cc70
--- /dev/null
+++ b/be/src/format_v2/expr/evolution_cast.h
@@ -0,0 +1,40 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "format_v2/expr/cast.h"
+
+namespace doris::format {
+
+// File-to-table conversions own external schema evolution validation; SQL casts do not.
+class EvolutionCast final : public Cast {
+ ENABLE_FACTORY_CREATOR(EvolutionCast);
+
+public:
+ explicit EvolutionCast(const DataTypePtr& type) : Cast(type) {}
+
+ Status prepare(RuntimeState* state, const RowDescriptor& desc, VExprContext* context) override;
+ std::string debug_string() const override;
+ Status clone_node(VExprSPtr* cloned_expr) const override {
+ DORIS_CHECK(cloned_expr != nullptr);
+ *cloned_expr = EvolutionCast::create_shared(_data_type);
+ return Status::OK();
+ }
+};
+
+} // namespace doris::format
diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp
index 615beaf233d..28bf76b2c0c 100644
--- a/be/src/format_v2/orc/orc_reader.cpp
+++ b/be/src/format_v2/orc/orc_reader.cpp
@@ -20,6 +20,7 @@
#include <cctz/time_zone.h>
#include <gen_cpp/Types_types.h>
+#include <Timezone.hh>
#include <algorithm>
#include <atomic>
#include <cctype>
@@ -62,11 +63,13 @@
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_timestamptz.h"
#include "core/data_type/data_type_uuid.h"
#include "core/data_type/data_type_varbinary.h"
#include "core/data_type_serde/data_type_serde.h"
#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
#include "core/types.h"
#include "core/value/timestamptz_value.h"
#include "core/value/vdatetime_value.h"
@@ -672,6 +675,11 @@ bool build_zone_map_from_orc_statistics(const ::orc::Type& type,
case ::orc::TypeKind::DATE:
return set_date_zone_map(statistics, zone_map);
case ::orc::TypeKind::TIMESTAMP: {
+ if (type.hasAttributeKey("iceberg.timestamp-unit") &&
+ type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
+ // The microsecond statistics carrier cannot safely prune nanosecond values.
+ return false;
+ }
// ORC stores timestamp statistics as wall-clock values in UTC coordinates. Restore them
// with UTC so the session timezone does not shift the civil time.
static const auto utc_time_zone = cctz::utc_time_zone();
@@ -741,6 +749,113 @@ Status find_projected_minmax_leaf(const ::orc::Type& root_type,
} // namespace
+// OrcReaderScanState has external linkage, so GCC requires its member types to stay
+// outside the anonymous namespace (-Wsubobject-linkage).
+// Keep SDK workarounds and their scan requirements in one plan. SerDes receive only the
+// resulting decode context and never need to infer which reader strategy produced a batch.
+class OrcTimestampReadPlan {
+public:
+ Status configure(const ::orc::Type& file_schema,
+ const std::vector<format::LocalColumnId>& read_columns,
+ const format::FileScanRequest& request,
+ const cctz::time_zone& session_timezone) {
+ _context = OrcTimestampDecodeContext(session_timezone);
+ _timezone_source = TimezoneSource::SESSION;
+ _timestamp_columns.clear();
+ if (!orc_serde_utils::has_nanosecond_timestamp(file_schema)) {
+ return Status::OK();
+ }
+ bool needs_writer_timezone = false;
+ for (const auto column_id : read_columns) {
+ if (is_virtual_column(column_id)) {
+ continue;
+ }
+ const auto& type = *file_schema.getSubtype(column_id.value());
+ const auto* projection = find_request_projection(request, column_id);
+ DORIS_CHECK(projection != nullptr);
+ std::set<uint64_t> projected_ids;
+ const auto* selected_ids = projection->project_all_children ? nullptr : &projected_ids;
+ if (selected_ids != nullptr) {
+ RETURN_IF_ERROR(collect_projected_type_ids(type, *projection, &projected_ids));
+ }
+ // A NANOS sibling affects the SDK only when a plain timestamp is decoded. Match
+ // includeTypes so unread nested leaves cannot trigger footer reads or dense decoding.
+ needs_writer_timezone |= has_timestamp(type, false, selected_ids);
+ if (has_timestamp(type, true, selected_ids)) {
+ _timestamp_columns.insert(column_id);
+ }
+ }
+ _timezone_source =
+ needs_writer_timezone ? TimezoneSource::STRIPE_WRITER : TimezoneSource::SESSION;
+ return Status::OK();
+ }
+
+ bool requires_stripe_reader() const {
+ return _timezone_source == TimezoneSource::STRIPE_WRITER;
+ }
+
+ bool requires_dense_decoding(format::LocalColumnId column_id) const {
+ // The pinned SDK's selected timestamp decoder checks a fraction at the wrong row index.
+ // Decode timestamp leaders densely while retaining lazy reads for other followers.
+ return requires_stripe_reader() && _timestamp_columns.contains(column_id);
+ }
+
+ Status configure_stripe(const std::string& writer_timezone, ::orc::WriterId writer,
+ ::orc::RowReaderOptions* options) {
+ DORIS_CHECK(requires_stripe_reader());
+ if (!TimezoneUtils::find_cctz_time_zone(writer_timezone, _context.reader_timezone)) {
+ return Status::InvalidArgument("Unknown ORC stripe writer timezone: {}",
+ writer_timezone);
+ }
+ // Matching zones preserves the SDK's UTC carrier until its negative-fraction correction
+ // is complete. SerDes then apply historical offsets to that original second with cctz.
+ options->setTimezoneName(writer_timezone);
+ if (writer == ::orc::ORC_JAVA_WRITER) {
+ // Java and C++ ancient carriers differ. Preserve the SDK Java writer rules
+ // instead of applying cctz LMT to a Java standard-offset carrier.
+ const auto* timezone = &::orc::getTimezoneByName(writer_timezone);
+ _context.writer_timezone = [timezone,
+ zones = std::unordered_map<int64_t, cctz::time_zone> {}](
+ int64_t seconds) mutable {
+ const auto offset = timezone->getVariant(seconds).gmtOffset;
+ auto it = zones.find(offset);
+ if (it == zones.end()) {
+ // Cache the few SDK offsets; fixed_time_zone takes a global cache lock.
+ it = zones.emplace(offset, cctz::fixed_time_zone(cctz::seconds(offset))).first;
+ }
+ return it->second;
+ };
+ }
+ return Status::OK();
+ }
+
+ const OrcTimestampDecodeContext& decode_context() const { return _context; }
+
+private:
+ enum class TimezoneSource { SESSION, STRIPE_WRITER };
+
+ static bool has_timestamp(const ::orc::Type& type, bool include_instant,
+ const std::set<uint64_t>* selected_ids) {
+ if (selected_ids != nullptr && !selected_ids->contains(type.getColumnId())) {
+ return false;
+ }
+ if (type.getKind() == ::orc::TIMESTAMP ||
+ (include_instant && type.getKind() == ::orc::TIMESTAMP_INSTANT)) {
+ return true;
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ if (has_timestamp(*type.getSubtype(child), include_instant, selected_ids)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ std::set<format::LocalColumnId> _timestamp_columns;
+ TimezoneSource _timezone_source = TimezoneSource::SESSION;
+ OrcTimestampDecodeContext _context {cctz::utc_time_zone()};
+};
+
class OrcReader::OrcFilterImpl final : public ::orc::ORCFilter {
public:
explicit OrcFilterImpl(OrcReader* reader) : _reader(reader) {}
@@ -763,6 +878,9 @@ struct OrcReaderScanState {
uint64_t offset = 0;
uint64_t length = 0;
bool disable_sarg = false;
+ uint64_t first_row = 0;
+ uint64_t end_row = 0;
+ std::optional<std::string> writer_timezone;
};
std::unique_ptr<::orc::Reader> reader;
@@ -773,6 +891,7 @@ struct OrcReaderScanState {
::orc::RowReaderOptions active_row_reader_options;
std::string timezone = TimezoneUtils::default_time_zone;
cctz::time_zone timezone_obj;
+ OrcTimestampReadPlan timestamp_plan;
std::unique_ptr<::orc::RowReader> row_reader;
const ::orc::Type* selected_type = nullptr;
std::unique_ptr<::orc::ColumnVectorBatch> batch;
@@ -798,7 +917,9 @@ struct OrcReaderScanState {
std::vector<uint64_t> decoded_date_column_ids;
std::vector<StripeRange> selected_stripe_ranges;
size_t current_stripe_range = 0;
- bool stripe_pruning_applied = false;
+ size_t next_stripe_range = 0;
+ size_t timestamp_stripe_window = 1;
+ bool stripe_ranges_applied = false;
size_t next_row_id = 0;
bool row_reader_created = false;
@@ -813,7 +934,6 @@ OrcReader::OrcReader(std::shared_ptr<io::FileSystemProperties>& system_propertie
: FileReader(system_properties, file_description, io_ctx, profile),
_global_rowid_context(std::move(global_rowid_context)),
_enable_mapping_timestamp_tz(enable_mapping_timestamp_tz),
- _enable_mapping_varbinary(enable_mapping_varbinary),
_file_context(std::move(file_context)) {}
OrcReader::~OrcReader() = default;
@@ -829,6 +949,10 @@ void OrcReader::_init_profile() {
file_scan_profile::ensure_hierarchy(_profile);
_orc_profile.total_time =
ADD_CHILD_TIMER_WITH_LEVEL(_profile, orc_profile, file_scan_profile::FILE_READER, 1);
+ _orc_profile.row_readers_created = ADD_CHILD_COUNTER_WITH_LEVEL(
+ _profile, "OrcRowReadersCreated", TUnit::UNIT, orc_profile, 1);
+ _orc_profile.timestamp_stripe_footers_read = ADD_CHILD_COUNTER_WITH_LEVEL(
+ _profile, "OrcTimestampStripeFootersRead", TUnit::UNIT, orc_profile, 1);
_orc_profile.reader_call =
ADD_CHILD_COUNTER_WITH_LEVEL(_profile, "ReaderCall", TUnit::UNIT, orc_profile, 1);
_orc_profile.reader_inclusive_latency_us = ADD_CHILD_COUNTER_WITH_LEVEL(
@@ -900,7 +1024,7 @@ void OrcReader::_collect_profile() const {
orc_metric_value(reader_metrics.IOBlockingLatencyUs));
int64_t selected_row_group_count = orc_metric_value(reader_metrics.SelectedRowGroupCount);
int64_t evaluated_row_group_count = orc_metric_value(reader_metrics.EvaluatedRowGroupCount);
- if (_state->stripe_pruning_applied) {
+ if (_reader_statistics.filtered_row_groups > 0) {
selected_row_group_count = _reader_statistics.read_row_groups;
evaluated_row_group_count =
_reader_statistics.filtered_row_groups + _reader_statistics.read_row_groups;
@@ -1091,9 +1215,16 @@ void OrcReader::set_condition_cache_context(std::shared_ptr<ConditionCacheContex
int64_t OrcReader::get_total_rows() const {
DORIS_CHECK(_state != nullptr);
- if (_state->stripe_pruning_applied && _state->selected_stripe_ranges.empty()) {
+ if (_state->stripe_ranges_applied && _state->selected_stripe_ranges.empty()) {
return 0;
}
+ if (_state->timestamp_plan.requires_stripe_reader() && _state->stripe_ranges_applied) {
+ uint64_t rows = 0;
+ for (const auto& range : _state->selected_stripe_ranges) {
+ rows += range.end_row - range.first_row;
+ }
+ return cast_set<int64_t>(rows);
+ }
if (_state->row_reader != nullptr) {
return cast_set<int64_t>(_state->row_reader_range_rows);
}
@@ -1131,14 +1262,14 @@ DataTypePtr OrcReader::_convert_to_doris_type(const ::orc::Type& type) const {
data_type = std::make_shared<DataTypeString>();
break;
case ::orc::TypeKind::BINARY:
- // A logical UUID remains a UUID when the raw-binary default is enabled.
- if (type.hasAttributeKey("doris.logical_type") &&
- type.getAttributeValue("doris.logical_type") == "uuid") {
+ // UUID annotations retain their native mapping; unannotated bytes never become text.
+ if ((type.hasAttributeKey("doris.logical_type") &&
+ type.getAttributeValue("doris.logical_type") == "uuid") ||
+ (type.hasAttributeKey("iceberg.binary-type") &&
+ type.getAttributeValue("iceberg.binary-type") == "UUID")) {
data_type = std::make_shared<DataTypeUUID>();
- } else if (_enable_mapping_varbinary) {
- data_type = std::make_shared<DataTypeVarbinary>();
} else {
- data_type = std::make_shared<DataTypeString>();
+ data_type = std::make_shared<DataTypeVarbinary>();
}
break;
case ::orc::TypeKind::VARCHAR:
@@ -1153,7 +1284,12 @@ DataTypePtr OrcReader::_convert_to_doris_type(const ::orc::Type& type) const {
data_type = std::make_shared<DataTypeDateV2>();
break;
case ::orc::TypeKind::TIMESTAMP:
- data_type = std::make_shared<DataTypeDateTimeV2>(6);
+ if (type.hasAttributeKey("iceberg.timestamp-unit") &&
+ type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
+ data_type = std::make_shared<DataTypeTimeStampNs>();
+ } else {
+ data_type = std::make_shared<DataTypeDateTimeV2>(6);
+ }
break;
case ::orc::TypeKind::TIMESTAMP_INSTANT:
if (_enable_mapping_timestamp_tz) {
@@ -1399,20 +1535,32 @@ Status OrcReader::open(std::shared_ptr<format::FileScanRequest> request) {
_apply_split_range();
RETURN_IF_ERROR(_select_stripe_ranges_by_statistics());
- if (_state->stripe_pruning_applied && _state->selected_stripe_ranges.empty()) {
+ RETURN_IF_ERROR(_prepare_timestamp_stripe_ranges());
+ if (_state->stripe_ranges_applied && _state->selected_stripe_ranges.empty()) {
_eof = true;
return Status::OK();
}
- _apply_current_stripe_range();
+ RETURN_IF_ERROR(_apply_current_stripe_range());
RETURN_IF_ERROR(_create_row_reader());
if (_request->row_ids.has_value()) {
+ size_t range_index = 0;
for (const int64_t row_id : *_request->row_ids) {
- if (static_cast<uint64_t>(row_id) < _state->row_reader_range_first_row ||
- static_cast<uint64_t>(row_id) >= _state->row_reader_range_end_row) {
+ uint64_t first_row = _state->row_reader_range_first_row;
+ uint64_t end_row = _state->row_reader_range_end_row;
+ if (_state->timestamp_plan.requires_stripe_reader()) {
+ const auto& ranges = _state->selected_stripe_ranges;
+ while (range_index + 1 < ranges.size() &&
+ std::cmp_greater_equal(row_id, ranges[range_index].end_row)) {
+ ++range_index;
+ }
+ first_row = ranges[range_index].first_row;
+ end_row = ranges[range_index].end_row;
+ }
+ if (std::cmp_less(row_id, first_row) || std::cmp_greater_equal(row_id, end_row)) {
return Status::InvalidArgument(
"ORC row id {} is outside the current split row range [{}, {})", row_id,
- _state->row_reader_range_first_row, _state->row_reader_range_end_row);
+ first_row, end_row);
}
}
}
@@ -1470,6 +1618,20 @@ Status OrcReader::_configure_row_reader_projection() {
DORIS_CHECK(projection != nullptr);
has_complex_projection = has_complex_projection || has_pruned_projection(*projection);
}
+ RETURN_IF_ERROR(_state->timestamp_plan.configure(*_state->root_type, _state->read_columns,
+ *_request, _state->timezone_obj));
+ auto filter_projections = _request->predicate_columns;
+ if (_state->orc_lazy_read_enabled) {
+ for (const auto column_id : _state->read_columns) {
+ if (!is_virtual_column(column_id) &&
+ _state->timestamp_plan.requires_dense_decoding(column_id) &&
+ std::ranges::none_of(filter_projections, [column_id](const auto& projection) {
+ return projection.column_id() == column_id;
+ })) {
+ filter_projections.push_back(*find_request_projection(*_request, column_id));
+ }
+ }
+ }
if (!has_complex_projection) {
std::list<uint64_t> include_columns;
for (const auto file_column_id : _state->read_columns) {
@@ -1481,7 +1643,7 @@ Status OrcReader::_configure_row_reader_projection() {
_state->row_reader_options.include(include_columns);
if (_state->orc_lazy_read_enabled) {
std::list<std::string> filter_columns;
- for (const auto& projection : _request->predicate_columns) {
+ for (const auto& projection : filter_projections) {
if (is_row_position_column(projection.column_id())) {
continue;
}
@@ -1521,8 +1683,8 @@ Status OrcReader::_configure_row_reader_projection() {
_state->row_reader_options.includeTypes(include_type_id_list);
if (_state->orc_lazy_read_enabled) {
std::set<uint64_t> filter_type_ids;
- RETURN_IF_ERROR(collect_lazy_filter_type_ids(
- *_state->root_type, _request->predicate_columns, &filter_type_ids));
+ RETURN_IF_ERROR(collect_lazy_filter_type_ids(*_state->root_type, filter_projections,
+ &filter_type_ids));
DORIS_CHECK(!filter_type_ids.empty());
std::list<uint64_t> filter_type_id_list(filter_type_ids.begin(), filter_type_ids.end());
_state->row_reader_options.filterTypes(filter_type_id_list);
@@ -1640,7 +1802,7 @@ Status OrcReader::_collect_split_stripes(std::vector<uint64_t>* stripe_indices)
Status OrcReader::_collect_selected_stripes(std::vector<uint64_t>* stripe_indices) const {
DORIS_CHECK(stripe_indices != nullptr);
stripe_indices->clear();
- if (!_state->stripe_pruning_applied) {
+ if (!_state->stripe_ranges_applied) {
return _collect_split_stripes(stripe_indices);
}
const auto stripe_count = _state->reader->getNumberOfStripes();
@@ -1722,7 +1884,7 @@ void OrcReader::_apply_split_range() {
Status OrcReader::_select_stripe_ranges_by_statistics() {
_state->selected_stripe_ranges.clear();
_state->current_stripe_range = 0;
- _state->stripe_pruning_applied = false;
+ _state->stripe_ranges_applied = false;
const bool has_search_argument = _state->row_reader_options.getSearchArgument() != nullptr;
if (!has_search_argument) {
return Status::OK();
@@ -1819,7 +1981,7 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
- _state->stripe_pruning_applied = true;
+ _state->stripe_ranges_applied = true;
_reader_statistics.filtered_row_groups = cast_set<int32_t>(filtered_stripes);
_reader_statistics.filtered_row_groups_by_min_max = cast_set<int32_t>(filtered_stripes);
_reader_statistics.filtered_group_rows = filtered_rows;
@@ -1843,6 +2005,7 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
.offset = offset,
.length = end_offset - offset,
.disable_sarg = unsafe_date_stripes.contains(first_stripe),
+ .writer_timezone = std::nullopt,
});
} catch (const std::exception& e) {
return Status::InternalError("Failed to build ORC stripe read range: {}", e.what());
@@ -1867,24 +2030,144 @@ Status OrcReader::_select_stripe_ranges_by_statistics() {
return Status::OK();
}
-void OrcReader::_apply_current_stripe_range() {
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty()) {
- return;
+Status OrcReader::_prepare_timestamp_stripe_ranges() {
+ if (!_state->timestamp_plan.requires_stripe_reader()) {
+ return Status::OK();
+ }
+ std::vector<uint64_t> stripes;
+ std::set<uint64_t> unsafe_date_stripes;
+ if (_state->stripe_ranges_applied) {
+ for (const auto& range : _state->selected_stripe_ranges) {
+ for (auto stripe = range.first_stripe; stripe < range.last_stripe; ++stripe) {
+ stripes.push_back(stripe);
+ if (range.disable_sarg) {
+ unsafe_date_stripes.insert(stripe);
+ }
+ }
+ }
+ } else {
+ RETURN_IF_ERROR(_collect_split_stripes(&stripes));
}
- DORIS_CHECK(_state->current_stripe_range < _state->selected_stripe_ranges.size());
- const auto& stripe_range = _state->selected_stripe_ranges[_state->current_stripe_range];
- _state->row_reader_options.range(stripe_range.offset, stripe_range.length);
+ _state->selected_stripe_ranges.clear();
+ _state->stripe_ranges_applied = true;
+ _reader_statistics.read_row_groups = cast_set<int32_t>(stripes.size());
+ if (stripes.empty()) {
+ return Status::OK();
+ }
+ try {
+ uint64_t first_row = 0;
+ size_t selected = 0;
+ for (uint64_t index = 0; index <= stripes.back(); ++index) {
+ const auto stripe = _state->reader->getStripe(index);
+ const auto end_row = first_row + stripe->getNumberOfRows();
+ if (index == stripes[selected]) {
+ // File-tail stripe descriptors are cheap. Defer footer I/O until a stripe
+ // participates in the current scan window (or an explicit row-id lookup).
+ _state->selected_stripe_ranges.push_back({
+ .first_stripe = index,
+ .last_stripe = index + 1,
+ .offset = stripe->getOffset(),
+ .length = stripe->getLength(),
+ .disable_sarg = unsafe_date_stripes.contains(index),
+ .first_row = first_row,
+ .end_row = end_row,
+ .writer_timezone = std::nullopt,
+ });
+ ++selected;
+ }
+ first_row = end_row;
+ }
+ } catch (const std::exception& e) {
+ return Status::InternalError("Failed to prepare ORC timestamp stripe ranges: {}", e.what());
+ }
+ return Status::OK();
+}
+
+Status OrcReader::_apply_current_stripe_range() {
+ if (!_state->stripe_ranges_applied || _state->selected_stripe_ranges.empty()) {
+ return Status::OK();
+ }
+ auto& ranges = _state->selected_stripe_ranges;
+ auto& current = _state->current_stripe_range;
+ DORIS_CHECK(current < ranges.size());
+ if (_state->timestamp_plan.requires_stripe_reader() && _request->row_ids.has_value() &&
+ _state->next_row_id < _request->row_ids->size()) {
+ const auto row_id = (*_request->row_ids)[_state->next_row_id];
+ while (current + 1 < ranges.size() &&
+ std::cmp_greater_equal(row_id, ranges[current].end_row)) {
+ ++current;
+ }
+ }
+ auto range = ranges[current];
+ _state->next_stripe_range = current + 1;
+ if (_state->timestamp_plan.requires_stripe_reader()) {
+ try {
+ const auto timezone_for = [&](size_t index) -> const std::string& {
+ auto& cached = ranges[index].writer_timezone;
+ if (!cached.has_value()) {
+ const auto stripe = _state->reader->getStripe(ranges[index].first_stripe);
+ const auto& name = stripe->getWriterTimezone();
+ cached = name.empty() ? "GMT" : name;
+ if (_profile != nullptr) {
+ COUNTER_UPDATE(_orc_profile.timestamp_stripe_footers_read, 1);
+ }
+ }
+ return *cached;
+ };
+ const auto& timezone = timezone_for(current);
+ // Start with one stripe for LIMIT/row-id latency. Exponentially growing windows
+ // amortize the SDK's whole-directory RowReader setup for long same-zone scans.
+ const size_t end =
+ current + std::min(_state->timestamp_stripe_window, ranges.size() - current);
+ while (_state->next_stripe_range < end) {
+ const auto next = _state->next_stripe_range;
+ const auto& candidate = ranges[next];
+ if (candidate.first_stripe != range.last_stripe ||
+ candidate.disable_sarg != range.disable_sarg) {
+ break;
+ }
+ if (_request->row_ids.has_value()) {
+ const auto& ids = *_request->row_ids;
+ const auto requested =
+ std::lower_bound(ids.begin(), ids.end(), candidate.first_row);
+ if (requested == ids.end() ||
+ std::cmp_greater_equal(*requested, candidate.end_row)) {
+ break;
+ }
+ }
+ if (timezone_for(next) != timezone) {
+ break;
+ }
+ range.last_stripe = candidate.last_stripe;
+ range.length = candidate.offset + candidate.length - range.offset;
+ range.end_row = candidate.end_row;
+ ++_state->next_stripe_range;
+ }
+ _state->timestamp_stripe_window +=
+ std::min(_state->timestamp_stripe_window,
+ ranges.size() - _state->timestamp_stripe_window);
+ RETURN_IF_ERROR(_state->timestamp_plan.configure_stripe(
+ timezone, _state->reader->getWriterId(), &_state->row_reader_options));
+ } catch (const std::exception& e) {
+ if (is_orc_stop(_io_ctx.get(), e)) {
+ return Status::EndOfFile("stop");
+ }
+ return Status::InternalError("Failed to read ORC stripe writer timezone: {}", e.what());
+ }
+ }
+ _state->row_reader_options.range(range.offset, range.length);
+ return Status::OK();
}
Status OrcReader::_advance_to_next_stripe_range(bool* advanced) {
DORIS_CHECK(advanced != nullptr);
*advanced = false;
- if (!_state->stripe_pruning_applied || _state->selected_stripe_ranges.empty() ||
- _state->current_stripe_range + 1 >= _state->selected_stripe_ranges.size()) {
+ if (!_state->stripe_ranges_applied || _state->selected_stripe_ranges.empty() ||
+ _state->next_stripe_range >= _state->selected_stripe_ranges.size()) {
return Status::OK();
}
- ++_state->current_stripe_range;
- _apply_current_stripe_range();
+ _state->current_stripe_range = _state->next_stripe_range;
+ RETURN_IF_ERROR(_apply_current_stripe_range());
RETURN_IF_ERROR(_create_row_reader());
*advanced = true;
return Status::OK();
@@ -1899,7 +2182,7 @@ Status OrcReader::_create_row_reader() {
_state->row_reader.reset();
auto& options = _state->active_row_reader_options;
options = _state->row_reader_options;
- if (_state->stripe_pruning_applied &&
+ if (_state->stripe_ranges_applied &&
_state->selected_stripe_ranges[_state->current_stripe_range].disable_sarg) {
// The SDK also prunes row groups; clearing only a copy lets later safe ranges keep
// their SARG while the retained unsafe range reaches DATE decoding.
@@ -1907,6 +2190,9 @@ Status OrcReader::_create_row_reader() {
}
_state->row_reader = _state->reader->createRowReader(
options, _state->orc_lazy_read_enabled ? _orc_filter.get() : nullptr);
+ if (_profile != nullptr) {
+ COUNTER_UPDATE(_orc_profile.row_readers_created, 1);
+ }
_state->selected_type = &_state->row_reader->getSelectedType();
DORIS_CHECK(_state->selected_type->getKind() == ::orc::TypeKind::STRUCT);
// Row-id fetch seeks before every read; a one-row batch preserves exact selection instead
@@ -2182,7 +2468,7 @@ Status OrcReader::_decode_column(const ::orc::Type& file_type, const ::orc::Type
view.batch = &batch;
view.rows = rows;
view.selected_rows = selected_rows;
- view.timezone = &_state->timezone_obj;
+ view.timestamp_context = &_state->timestamp_plan.decode_context();
view.enable_mapping_timestamp_tz = _enable_mapping_timestamp_tz;
return column_type->get_serde()->read_column_from_orc(*column, view);
}
@@ -2221,6 +2507,13 @@ Status OrcReader::get_block(Block* file_block, size_t* rows, bool* eof) {
return Status::OK();
}
fetched_row_id = static_cast<uint64_t>((*_request->row_ids)[_state->next_row_id]);
+ // Row ids are sorted globally and can cross the per-writer-timezone ranges.
+ while (_state->timestamp_plan.requires_stripe_reader() &&
+ *fetched_row_id >= _state->row_reader_range_end_row) {
+ bool advanced = false;
+ RETURN_IF_ERROR(_advance_to_next_stripe_range(&advanced));
+ DORIS_CHECK(advanced);
+ }
_state->row_reader->seekToRow(*fetched_row_id);
}
// Condition-cache seeks can perform I/O, so keep them in the same cancellation
@@ -2388,6 +2681,11 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque
const auto* count_type =
_state->root_type->getSubtype(static_cast<uint64_t>(count_projection.local_id()));
DORIS_CHECK(count_type != nullptr);
+ // ORC timestamps can exceed signed nanos, including nested leaves. Millisecond
+ // statistics cannot replace the range validation performed during materialization.
+ if (orc_serde_utils::has_nanosecond_timestamp(*count_type)) {
+ return Status::NotSupported("ORC NANOS COUNT requires value validation");
+ }
std::vector<uint32_t> date_column_ids;
const auto collect_dates = [&](auto&& self, const ::orc::Type& type) -> void {
@@ -2489,6 +2787,25 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque
}
std::unique_ptr<::orc::StripeStatistics> stripe_statistics;
try {
+ if (leaf_type->getKind() == ::orc::TIMESTAMP &&
+ orc_serde_utils::has_nanosecond_timestamp(*_state->root_type) &&
+ _state->reader->getWriterId() == ::orc::ORC_CPP_WRITER) {
+ // Named-zone C++ statistics use SDK historical offsets, whereas rows use cctz.
+ // Preserve exact metadata aggregates for fixed-zone writers; otherwise scan rows.
+ const auto stripe = _state->reader->getStripe(stripe_index);
+ const auto& name = stripe->getWriterTimezone();
+ if (_profile != nullptr) {
+ COUNTER_UPDATE(_orc_profile.timestamp_stripe_footers_read, 1);
+ }
+ cctz::time_zone writer_zone;
+ int32_t offset = 0;
+ if (!TimezoneUtils::find_cctz_time_zone(name.empty() ? "GMT" : name,
+ writer_zone) ||
+ !TimezoneUtils::try_get_fixed_offset_seconds(writer_zone, &offset)) {
+ return Status::NotSupported(
+ "ORC C++ named-zone mixed nanos timestamps require row min/max");
+ }
+ }
stripe_statistics = _state->reader->getStripeStatistics(stripe_index);
} catch (const std::exception& e) {
return Status::InternalError("Failed to read ORC stripe statistics {}: {}",
diff --git a/be/src/format_v2/orc/orc_reader.h b/be/src/format_v2/orc/orc_reader.h
index 662509e25ad..42c166f2bdf 100644
--- a/be/src/format_v2/orc/orc_reader.h
+++ b/be/src/format_v2/orc/orc_reader.h
@@ -92,6 +92,8 @@ public:
private:
struct OrcProfile {
+ RuntimeProfile::Counter* row_readers_created = nullptr;
+ RuntimeProfile::Counter* timestamp_stripe_footers_read = nullptr;
RuntimeProfile::Counter* total_time = nullptr;
RuntimeProfile::Counter* reader_call = nullptr; // ReaderCall
RuntimeProfile::Counter* reader_inclusive_latency_us = nullptr; // ReaderInclusiveLatencyUs
@@ -150,7 +152,8 @@ private:
// stripes. Skips the call for a whole-file window to keep ORC library defaults.
void _apply_split_range();
Status _select_stripe_ranges_by_statistics();
- void _apply_current_stripe_range();
+ Status _prepare_timestamp_stripe_ranges();
+ Status _apply_current_stripe_range();
Status _advance_to_next_stripe_range(bool* advanced);
Status _create_row_reader();
@@ -202,7 +205,6 @@ private:
OrcProfile _orc_profile; // RuntimeProfile counters
std::optional<format::GlobalRowIdContext> _global_rowid_context;
bool _enable_mapping_timestamp_tz = false;
- bool _enable_mapping_varbinary = false;
std::shared_ptr<const FileContext> _file_context;
};
diff --git a/be/src/format_v2/parquet/native_schema_desc.cpp b/be/src/format_v2/parquet/native_schema_desc.cpp
index 97ebfe4aacc..0b3ad90e926 100644
--- a/be/src/format_v2/parquet/native_schema_desc.cpp
+++ b/be/src/format_v2/parquet/native_schema_desc.cpp
@@ -853,16 +853,9 @@ std::pair<DataTypePtr, bool> NativeFieldDescriptor::get_doris_type(
ans.first = DataTypeFactory::instance().create_data_type(TYPE_DOUBLE, nullable);
break;
case tparquet::Type::BYTE_ARRAY:
- if (_enable_mapping_varbinary) {
- // if physical_schema not set logicalType and converted_type,
- // we treat BYTE_ARRAY as VARBINARY by default, so that we can read all data directly.
- ans.first = DataTypeFactory::instance().create_data_type(TYPE_VARBINARY, nullable);
- } else {
- ans.first = DataTypeFactory::instance().create_data_type(TYPE_STRING, nullable);
- }
- break;
case tparquet::Type::FIXED_LEN_BYTE_ARRAY:
- ans.first = DataTypeFactory::instance().create_data_type(TYPE_STRING, nullable);
+ // Text annotations are handled above; unannotated bytes always retain binary semantics.
+ ans.first = DataTypeFactory::instance().create_data_type(TYPE_VARBINARY, nullable);
break;
default:
throw Exception(Status::InternalError("Not supported parquet logicalType{}",
@@ -915,6 +908,11 @@ std::pair<DataTypePtr, bool> NativeFieldDescriptor::convert_to_doris_type(
// sub-second values are silently truncated by the target SerDe.
ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMEV2, nullable, 0, scale);
} else if (logicalType.__isset.TIMESTAMP) {
+ if (logicalType.TIMESTAMP.unit.__isset.NANOS && !logicalType.TIMESTAMP.isAdjustedToUTC) {
+ // Materialize nanoseconds before any table cast; DATETIMEV2 would discard the remainder.
+ ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMESTAMP_NS, nullable);
+ return ans;
+ }
if (_enable_mapping_timestamp_tz) {
if (logicalType.TIMESTAMP.isAdjustedToUTC) {
// treat TIMESTAMP with isAdjustedToUTC as TIMESTAMPTZ
@@ -927,10 +925,8 @@ std::pair<DataTypePtr, bool> NativeFieldDescriptor::convert_to_doris_type(
ans.first = DataTypeFactory::instance().create_data_type(
TYPE_DATETIMEV2, nullable, 0, logicalType.TIMESTAMP.unit.__isset.MILLIS ? 3 : 6);
} else if (logicalType.__isset.UUID) {
- // UUID is a logical type, so the raw-binary default must not override it.
- // Preserve table-requested raw carriers for Iceberg equality deletes and defaults.
- auto uuid_type = _preserve_binary_uuid ? TYPE_STRING : TYPE_UUID;
- ans.first = DataTypeFactory::instance().create_data_type(uuid_type, nullable, -1, -1, 16);
+ // UUID annotations describe a logical UUID even when binary mapping is enabled.
+ ans.first = DataTypeFactory::instance().create_data_type(TYPE_UUID, nullable);
} else if (logicalType.__isset.FLOAT16) {
ans.first = DataTypeFactory::instance().create_data_type(TYPE_FLOAT, nullable);
} else {
diff --git a/be/src/format_v2/parquet/native_schema_desc.h b/be/src/format_v2/parquet/native_schema_desc.h
index 1f788bef999..ff4d72aca05 100644
--- a/be/src/format_v2/parquet/native_schema_desc.h
+++ b/be/src/format_v2/parquet/native_schema_desc.h
@@ -104,9 +104,9 @@ private:
std::unordered_map<std::string, NativeFieldSchema*> _name_to_field;
// Used in from_thrift, marking the next schema position that should be parsed
size_t _next_schema_pos;
- // useful for parse_node_field to decide whether to convert byte_array to VARBINARY type
- bool _enable_mapping_varbinary = false;
bool _enable_mapping_timestamp_tz = false;
+ bool _enable_mapping_varbinary =
+ false; // Legacy input; native binary and UUID mapping is unconditional.
bool _preserve_binary_uuid = false;
private:
@@ -188,6 +188,7 @@ public:
void assign_ids();
const NativeFieldSchema* find_column_by_id(uint64_t column_id) const;
+ // Ordinary binary leaves ignore this option; UUID retains its explicit binary override.
void set_enable_mapping_varbinary(bool enable) { _enable_mapping_varbinary = enable; }
void set_enable_mapping_timestamp_tz(bool enable) { _enable_mapping_timestamp_tz = enable; }
void set_preserve_binary_uuid(bool preserve) { _preserve_binary_uuid = preserve; }
diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp
index 659a41a80c7..0379327486c 100644
--- a/be/src/format_v2/parquet/parquet_reader.cpp
+++ b/be/src/format_v2/parquet/parquet_reader.cpp
@@ -531,8 +531,14 @@ DataTypePtr apply_projection_timestamp_semantics(ParquetColumnSchema* column_sch
const bool physical_timestamp =
descriptor.physical_type == tparquet::Type::INT96 || descriptor.is_timestamp;
if (physical_timestamp && projection.timestamp_is_adjusted_to_utc.has_value()) {
- const auto target =
- *projection.timestamp_is_adjusted_to_utc ? TYPE_TIMESTAMPTZ : TYPE_DATETIMEV2;
+ // A table's wall-clock annotation must not narrow a nanosecond physical column.
+ auto target = TYPE_DATETIMEV2;
+ if (*projection.timestamp_is_adjusted_to_utc) {
+ target = TYPE_TIMESTAMPTZ;
+ } else if (column_schema->type != nullptr &&
+ column_schema->type->get_primitive_type() == TYPE_TIMESTAMP_NS) {
+ target = TYPE_TIMESTAMP_NS;
+ }
column_schema->type = DataTypeFactory::instance().create_data_type(
target, column_schema->type != nullptr && column_schema->type->is_nullable(), 0,
timestamp_tz_scale(descriptor));
diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h
index 15a10a2f265..d5830b91e6a 100644
--- a/be/src/format_v2/parquet/parquet_reader.h
+++ b/be/src/format_v2/parquet/parquet_reader.h
@@ -104,7 +104,8 @@ private:
std::optional<format::GlobalRowIdContext> _global_rowid_context; // global RowId context
size_t _batch_size = ParquetScanScheduler::DEFAULT_READ_BATCH_SIZE;
bool _enable_mapping_timestamp_tz = false; // whether UTC timestamps are mapped to TIMESTAMPTZ
- bool _enable_mapping_varbinary = false; // whether raw BYTE_ARRAY is mapped to VARBINARY
+ bool _enable_mapping_varbinary =
+ false; // UUID override only; ordinary V2 binary mapping is unconditional.
std::shared_ptr<const FileContext> _file_context;
int64_t _format_split_id = -1;
int64_t _format_split_id_end = -1;
diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp
index 0e0206aa0a9..068cdbf03c1 100644
--- a/be/src/format_v2/parquet/parquet_scan.cpp
+++ b/be/src/format_v2/parquet/parquet_scan.cpp
@@ -1941,6 +1941,7 @@ bool get_typed_dictionary_raw_values(PrimitiveType primitive_type, const IColumn
GET_TYPED_DICTIONARY_VALUES(TYPE_DATEV2);
GET_TYPED_DICTIONARY_VALUES(TYPE_DATETIMEV2);
GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMPTZ);
+ GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMP_NS);
GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMAL32);
GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMAL64);
GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMALV2);
diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp b/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp
index 43f659120c9..ac2b670da7b 100644
--- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp
+++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp
@@ -587,6 +587,8 @@ bool visit_nullable_expandable_column(IColumn& column, Visitor&& visitor) {
VISIT_COLUMN(ColumnUUID)
VISIT_COLUMN(ColumnTimeV2)
VISIT_COLUMN(ColumnTimeStampTz)
+ // Nanos columns also support in-place NULL expansion; do not decode once per null run.
+ VISIT_COLUMN(ColumnTimeStampNs)
VISIT_COLUMN(ColumnOffset32)
VISIT_COLUMN(ColumnOffset64)
VISIT_COLUMN(ColumnDecimal32)
diff --git a/be/src/format_v2/schema_reader.cpp b/be/src/format_v2/schema_reader.cpp
new file mode 100644
index 00000000000..f9860a86266
--- /dev/null
+++ b/be/src/format_v2/schema_reader.cpp
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/schema_reader.h"
+
+#include "format_v2/orc/orc_reader.h"
+#include "format_v2/parquet/parquet_reader.h"
+#include "io/file_factory.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::format {
+
+Status fetch_file_schema(const TFileScanRangeParams& params, const TFileRangeDesc& range,
+ const std::shared_ptr<io::IOContext>& io_ctx, RuntimeProfile* profile,
+ std::vector<ColumnDefinition>* schema) {
+ auto properties = std::make_shared<io::FileSystemProperties>();
+ // Schema RPCs carry the filesystem kind on the range, not necessarily in scan params.
+ properties->system_type = range.__isset.file_type ? range.file_type : params.file_type;
+ properties->properties = params.properties;
+ properties->hdfs_params = params.hdfs_params;
+ properties->broker_addresses = params.broker_addresses;
+ auto file = std::make_unique<io::FileDescription>();
+ file->path = range.path;
+ file->file_size = range.__isset.file_size ? range.file_size : -1;
+ file->mtime = range.__isset.modification_time ? range.modification_time : 0;
+ file->fs_name = range.fs_name;
+ if (range.__isset.file_cache_admission) {
+ file->file_cache_admission = range.file_cache_admission;
+ }
+ const bool timestamp_tz =
+ params.__isset.enable_mapping_timestamp_tz && params.enable_mapping_timestamp_tz;
+ const bool varbinary =
+ params.__isset.enable_mapping_varbinary && params.enable_mapping_varbinary;
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ std::unique_ptr<FileReader> reader;
+ // TVF targets must retain file precision before column mapping; a microsecond schema
+ // irreversibly rounds nanos even when the subsequent row scan uses the V2 reader.
+ if (params.format_type == TFileFormatType::FORMAT_PARQUET) {
+ reader = std::make_unique<parquet::ParquetReader>(properties, file, io_ctx, profile,
+ std::nullopt, timestamp_tz, varbinary);
+ } else if (params.format_type == TFileFormatType::FORMAT_ORC) {
+ // Preserve the UUID override while ordinary ORC BINARY always maps to VARBINARY.
+ reader = std::make_unique<orc::OrcReader>(properties, file, io_ctx, profile, std::nullopt,
+ timestamp_tz, varbinary);
+ } else {
+ return Status::NotSupported("V2 schema inference requires Parquet or ORC");
+ }
+ RETURN_IF_ERROR(reader->init(&state));
+ RETURN_IF_ERROR(reader->get_schema(schema));
+ return reader->close();
+}
+
+} // namespace doris::format
diff --git a/be/src/format_v2/schema_reader.h b/be/src/format_v2/schema_reader.h
new file mode 100644
index 00000000000..ad92c557ec9
--- /dev/null
+++ b/be/src/format_v2/schema_reader.h
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+#include <vector>
+
+#include "common/status.h"
+
+namespace doris {
+class RuntimeProfile;
+class TFileScanRangeParams;
+class TFileRangeDesc;
+namespace io {
+struct IOContext;
+}
+namespace format {
+struct ColumnDefinition;
+
+Status fetch_file_schema(const TFileScanRangeParams& params, const TFileRangeDesc& range,
+ const std::shared_ptr<io::IOContext>& io_ctx, RuntimeProfile* profile,
+ std::vector<ColumnDefinition>* schema);
+} // namespace format
+} // namespace doris
diff --git a/be/src/format_v2/table/iceberg_reader.cpp b/be/src/format_v2/table/iceberg_reader.cpp
index 96630e09cd1..88eb497c3dc 100644
--- a/be/src/format_v2/table/iceberg_reader.cpp
+++ b/be/src/format_v2/table/iceberg_reader.cpp
@@ -51,8 +51,8 @@
#include "exprs/vslot_ref.h"
#include "format/table/deletion_vector_reader.h"
#include "format/table/iceberg_default_value.h"
-#include "format_v2/expr/cast.h"
#include "format_v2/expr/equality_delete_predicate.h"
+#include "format_v2/expr/evolution_cast.h"
#include "format_v2/orc/orc_reader.h"
#include "format_v2/parquet/parquet_reader.h"
#include "format_v2/parquet/reader/column_reader.h"
@@ -597,7 +597,7 @@ static Status build_missing_equality_delete_key_expr(const format::ColumnDefinit
*key_expr = std::move(literal);
return Status::OK();
}
- auto cast_expr = Cast::create_shared(delete_key_type);
+ auto cast_expr = EvolutionCast::create_shared(delete_key_type);
cast_expr->add_child(literal);
*key_expr = std::move(cast_expr);
return Status::OK();
@@ -1447,7 +1447,7 @@ Status IcebergTableReader::_build_missing_equality_delete_key_expr(
DORIS_CHECK(table_leaf != nullptr);
DORIS_CHECK(table_leaf->type != nullptr);
if (!table_leaf->type->equals(*filter.key_types[key_idx])) {
- auto cast_expr = Cast::create_shared(filter.key_types[key_idx]);
+ auto cast_expr = EvolutionCast::create_shared(filter.key_types[key_idx]);
cast_expr->add_child(default_expr);
default_expr = std::move(cast_expr);
}
@@ -1501,7 +1501,7 @@ Status IcebergTableReader::_append_equality_delete_predicates(format::FileScanRe
if (field->type->equals(*filter.key_types[idx])) {
delete_predicate->add_child(key_expr);
} else {
- auto cast_expr = Cast::create_shared(filter.key_types[idx]);
+ auto cast_expr = EvolutionCast::create_shared(filter.key_types[idx]);
cast_expr->add_child(key_expr);
delete_predicate->add_child(cast_expr);
}
@@ -1769,10 +1769,9 @@ Status IcebergTableReader::_load_equality_delete_file(const TIcebergDeleteFileDe
path, request->local_positions.at(root_column_id).value(), &key_expr));
const auto& key_type = result->key_types[index];
if (!key_expr->data_type()->equals(*key_type)) {
- // Historical delete values are promoted into the comparison domain. For an
- // Iceberg-legal type promotion this cast is always widening, so it cannot turn a
- // non-NULL value into NULL.
- auto cast_expr = Cast::create_shared(key_type);
+ // Historical delete values share the promoted comparison domain. DATE-to-nanos
+ // must report overflow rather than turn a non-NULL equality key into NULL.
+ auto cast_expr = EvolutionCast::create_shared(key_type);
cast_expr->add_child(key_expr);
key_expr = std::move(cast_expr);
}
diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h
index 75f83de3ffb..2c3879cb198 100644
--- a/be/src/format_v2/table_reader.h
+++ b/be/src/format_v2/table_reader.h
@@ -57,8 +57,8 @@
#include "format/table/deletion_vector.h"
#include "format_v2/column_data.h"
#include "format_v2/column_mapper.h"
-#include "format_v2/expr/cast.h"
#include "format_v2/expr/delete_predicate.h"
+#include "format_v2/expr/evolution_cast.h"
#include "format_v2/file_reader.h"
#include "format_v2/file_scan_context.h"
#include "format_v2/parquet/reader/column_reader.h"
@@ -1354,7 +1354,7 @@ protected:
Block cast_block;
cast_block.insert({*column, input_type, column_name});
auto slot_ref = VSlotRef::create_shared(0, 0, -1, input_type, column_name);
- auto cast_expr = Cast::create_shared(cast_type);
+ auto cast_expr = EvolutionCast::create_shared(cast_type);
cast_expr->add_child(std::move(slot_ref));
auto cast_ctx = VExprContext::create_shared(std::move(cast_expr));
RowDescriptor row_desc;
diff --git a/be/src/service/internal_service.cpp b/be/src/service/internal_service.cpp
index 219b05258e7..fb297867f7f 100644
--- a/be/src/service/internal_service.cpp
+++ b/be/src/service/internal_service.cpp
@@ -76,9 +76,9 @@
#include "format/csv/csv_reader.h"
#include "format/generic_reader.h"
#include "format/json/new_json_reader.h"
-#include "format/orc/vorc_reader.h"
-#include "format/parquet/vparquet_reader.h"
#include "format/text/text_reader.h"
+#include "format_v2/column_data.h"
+#include "format_v2/schema_reader.h"
#include "format_v2/table/lance_reader.h"
#include "io/fs/local_file_system.h"
#include "io/fs/stream_load_pipe.h"
@@ -883,13 +883,19 @@ void PInternalService::fetch_table_schema(google::protobuf::RpcController* contr
file_slots, fetch_schema_batch_size, io_ctx.get());
break;
}
- case TFileFormatType::FORMAT_PARQUET: {
- reader = ParquetReader::create_unique(params, range, io_ctx, nullptr);
- break;
- }
+ case TFileFormatType::FORMAT_PARQUET:
case TFileFormatType::FORMAT_ORC: {
- reader = OrcReader::create_unique(params, range, fetch_schema_batch_size, "", io_ctx);
- break;
+ std::vector<format::ColumnDefinition> schema;
+ st = format::fetch_file_schema(params, range, io_ctx, profile.get(), &schema);
+ if (st.ok()) {
+ result->set_column_nums(schema.size());
+ for (const auto& column : schema) {
+ result->add_column_names(column.name);
+ column.type->to_protobuf(result->add_column_types());
+ }
+ }
+ st.to_protobuf(result->mutable_status());
+ return;
}
case TFileFormatType::FORMAT_JSON: {
reader = NewJsonReader::create_unique(profile.get(), params, range, file_slots,
diff --git a/be/test/core/data_type/data_type_timestamp_ns_test.cpp b/be/test/core/data_type/data_type_timestamp_ns_test.cpp
index aed7beb10a0..3801bbc79f8 100644
--- a/be/test/core/data_type/data_type_timestamp_ns_test.cpp
+++ b/be/test/core/data_type/data_type_timestamp_ns_test.cpp
@@ -23,6 +23,8 @@
#include <cstring>
#include <limits>
#include <memory>
+#include <orc/Type.hh>
+#include <orc/Vector.hh>
#include <string>
#include <string_view>
#include <vector>
@@ -34,6 +36,7 @@
#include "core/data_type/data_type_date_or_datetime_v2.h"
#include "core/data_type_serde/data_type_serde.h"
#include "core/data_type_serde/data_type_timestamp_ns_serde.h"
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
#include "core/string_buffer.hpp"
#include "core/value/vdatetime_value.h"
#include "exprs/function/cast/cast_parameters.h"
@@ -464,16 +467,18 @@ TEST(DataTypeTimeStampNsTest, SerDeRoundTripsTextProtobufAndBinary) {
EXPECT_EQ(binary_data, source_data);
}
-TEST(DataTypeTimeStampNsTest, SerDeExplicitlyRejectsUnsupportedOrcIo) {
+TEST(DataTypeTimeStampNsTest, SerDeWritesEmptyOrcBatch) {
const DataTypeTimeStampNs type;
const auto serde = type.get_serde();
auto column = type.create_column();
Arena arena;
DataTypeSerDe::FormatOptions options;
+ orc::TimestampVectorBatch batch(0, *orc::getDefaultPool());
const auto orc_write_status =
- serde->write_column_to_orc("UTC", *column, nullptr, nullptr, 0, 0, arena, options);
- EXPECT_TRUE(orc_write_status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) << orc_write_status;
+ serde->write_column_to_orc("UTC", *column, nullptr, &batch, 0, 0, arena, options);
+ EXPECT_TRUE(orc_write_status.ok()) << orc_write_status;
+ EXPECT_EQ(0, batch.numElements);
}
TEST(DataTypeTimeStampNsTest, DataTypeLiteralField) {
@@ -682,4 +687,37 @@ TEST(DataTypeTimeStampNsTest, SerDeBatchRejectsMultipleFractionalSeparators) {
.ok());
}
+TEST(DataTypeTimeStampNsTest, OrcRoundTripPreservesNanoseconds) {
+ DataTypeTimeStampNs type;
+ auto serde = type.get_serde();
+ auto input = type.create_column();
+ auto& values = assert_cast<ColumnTimeStampNs&>(*input).get_data();
+ for (int64_t nanos : {-1000000001LL, 0LL, 1LL, 123456789LL}) {
+ values.emplace_back(nanos);
+ }
+ orc::TimestampVectorBatch batch(4, *orc::getDefaultPool());
+ for (size_t i = 0; i < 4; ++i) {
+ batch.notNull[i] = 1;
+ }
+ Arena arena;
+ DataTypeSerDe::FormatOptions options;
+ ASSERT_TRUE(
+ serde->write_column_to_orc("UTC", *input, nullptr, &batch, 0, 4, arena, options).ok());
+ auto orc_type = orc::createPrimitiveType(orc::TIMESTAMP);
+ auto output = type.create_column();
+ auto timezone = cctz::utc_time_zone();
+ OrcDecodedColumnView view;
+ view.file_type = orc_type.get();
+ view.batch = &batch;
+ view.rows = 4;
+ const OrcTimestampDecodeContext timestamp_context(timezone);
+ view.timestamp_context = ×tamp_context;
+ ASSERT_TRUE(serde->read_column_from_orc(*output, view).ok());
+ const auto& result = assert_cast<const ColumnTimeStampNs&>(*output).get_data();
+ ASSERT_EQ(result.size(), values.size());
+ for (size_t i = 0; i < result.size(); ++i) {
+ EXPECT_EQ(result[i].epoch_nanos(), values[i].epoch_nanos());
+ }
+}
+
} // namespace doris
diff --git a/be/test/core/data_type_serde/data_type_serde_parquet_test.cpp b/be/test/core/data_type_serde/data_type_serde_parquet_test.cpp
index c6ea8c24ebd..3d49af3bd5c 100644
--- a/be/test/core/data_type_serde/data_type_serde_parquet_test.cpp
+++ b/be/test/core/data_type_serde/data_type_serde_parquet_test.cpp
@@ -33,6 +33,7 @@
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_time.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_timestamptz.h"
#include "core/data_type/data_type_uuid.h"
#include "core/data_type/data_type_varbinary.h"
@@ -1150,4 +1151,56 @@ TEST(DataTypeSerDeParquetTest, UuidPlainAndDictionaryPreserveBigEndianBits) {
}
} // namespace
+TEST(DataTypeSerDeParquetTest, TimestampNsPreservesSignedNanosecondsAndDictionary) {
+ DataTypeTimeStampNs type;
+ auto serde = type.get_serde();
+ ParquetDecodeContext context;
+ context.physical_type = ParquetPhysicalType::INT64;
+ context.time_unit = ParquetTimeUnit::NANOS;
+ const std::vector<int64_t> values = {std::numeric_limits<int64_t>::min(), -1, 0, 1, 123456789,
+ std::numeric_limits<int64_t>::max()};
+ for (bool dictionary : {false, true}) {
+ TestParquetDecodeSource source;
+ if (dictionary) {
+ source.set_fixed_dictionary<int64_t>(values, {0, 1, 2, 3, 4, 5});
+ context.encoding = ParquetValueEncoding::DICTIONARY;
+ } else {
+ source.set_fixed_values<int64_t>(values);
+ }
+ auto column = type.create_column();
+ ParquetMaterializationState state;
+ ASSERT_TRUE(serde->read_column_from_parquet(*column, source, context, values.size(), state)
+ .ok());
+ const auto& data = assert_cast<const ColumnTimeStampNs&>(*column).get_data();
+ ASSERT_EQ(data.size(), values.size());
+ for (size_t i = 0; i < values.size(); ++i) {
+ EXPECT_EQ(data[i].epoch_nanos(), values[i]);
+ }
+ }
+}
+
+TEST(DataTypeSerDeParquetTest, TimestampNsReadsInt96AndRejectsOverflow) {
+ DataTypeTimeStampNs type;
+ auto serde = type.get_serde();
+ ParquetDecodeContext context;
+ context.physical_type = ParquetPhysicalType::INT96;
+ TestParquetDecodeSource source;
+ source.set_fixed_values<TestParquetInt96Timestamp>(
+ {{.nanos_of_day = NANOS_PER_DAY - 1, .julian_day = JULIAN_UNIX_EPOCH - 1},
+ {.nanos_of_day = 1, .julian_day = JULIAN_UNIX_EPOCH}});
+ auto column = type.create_column();
+ ParquetMaterializationState state;
+ ASSERT_TRUE(serde->read_column_from_parquet(*column, source, context, 2, state).ok());
+ const auto& data = assert_cast<const ColumnTimeStampNs&>(*column).get_data();
+ EXPECT_EQ(data[0].epoch_nanos(), -1);
+ EXPECT_EQ(data[1].epoch_nanos(), 1);
+
+ TestParquetDecodeSource overflow;
+ overflow.set_fixed_values<int64_t>({std::numeric_limits<int64_t>::max()});
+ context.physical_type = ParquetPhysicalType::INT64;
+ context.time_unit = ParquetTimeUnit::MICROS;
+ EXPECT_FALSE(serde->read_column_from_parquet(*column, overflow, context, 1, state).ok());
+ EXPECT_EQ(column->size(), 2);
+}
+
} // namespace doris
diff --git a/be/test/core/data_type_serde/data_type_serde_varbinary_test.cpp b/be/test/core/data_type_serde/data_type_serde_varbinary_test.cpp
index 5abe3535f6f..700dda99a15 100644
--- a/be/test/core/data_type_serde/data_type_serde_varbinary_test.cpp
+++ b/be/test/core/data_type_serde/data_type_serde_varbinary_test.cpp
@@ -667,4 +667,36 @@ TEST_F(DataTypeVarbinarySerDeTest, DeserializeOneCellFromJsonWithRawBytes) {
}
}
+TEST_F(DataTypeVarbinarySerDeTest, OrcBinaryPreservesSelectedNullAndEmptyValues) {
+ auto type = ::orc::createPrimitiveType(::orc::BINARY);
+ ::orc::StringVectorBatch batch(4, *::orc::getDefaultPool());
+ batch.numElements = 4;
+ batch.hasNulls = true;
+ std::string bytes("a\0b\xff", 4);
+ for (size_t i = 0; i < 4; ++i) {
+ batch.notNull[i] = i != 1;
+ batch.data[i] = bytes.data();
+ batch.length[i] = bytes.size();
+ }
+ batch.data[1] = nullptr;
+ batch.length[1] = -1; // NULL payloads must never be inspected.
+ batch.data[2] = nullptr;
+ batch.length[2] = 0;
+ std::vector<size_t> selected {3, 1, 2, 0};
+ OrcDecodedColumnView view {
+ .file_type = type.get(), .batch = &batch, .rows = 4, .selected_rows = &selected};
+ DataTypeVarbinarySerDe serde;
+ auto column = ColumnVarbinary::create();
+ for (int repeat = 0; repeat < 2; ++repeat) {
+ ASSERT_TRUE(serde.read_column_from_orc(*column, view).ok());
+ }
+ ASSERT_EQ(column->size(), 8);
+ for (size_t i = 0; i < column->size(); ++i) {
+ EXPECT_EQ(column->get_data_at(i).to_string(), (i % 4 == 0 || i % 4 == 3) ? bytes : "");
+ }
+ batch.length[3] = -1;
+ auto invalid = ColumnVarbinary::create();
+ EXPECT_FALSE(serde.read_column_from_orc(*invalid, view).ok());
+}
+
} // namespace doris
diff --git a/be/test/core/data_type_serde/orc_timestamp_decode_context_test.cpp b/be/test/core/data_type_serde/orc_timestamp_decode_context_test.cpp
new file mode 100644
index 00000000000..0aa72360c15
--- /dev/null
+++ b/be/test/core/data_type_serde/orc_timestamp_decode_context_test.cpp
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
+
+#include <gtest/gtest.h>
+
+#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_timestamp_ns.h"
+#include "core/data_type_serde/data_type_serde.h"
+#include "core/data_type_serde/orc_serde_utils.h"
+
+namespace doris {
+
+TEST(OrcTimestampDecodeContextTest, SessionIsTheDefaultReaderTimezone) {
+ const OrcTimestampDecodeContext context(cctz::fixed_time_zone(cctz::seconds(8 * 3600)));
+ const auto plain = orc::createPrimitiveType(orc::TIMESTAMP);
+ const auto instant = orc::createPrimitiveType(orc::TIMESTAMP_INSTANT);
+ EXPECT_EQ(context.timezone_for(*plain).name(), context.session_timezone.name());
+ EXPECT_EQ(context.timezone_for(*instant).name(), context.session_timezone.name());
+}
+
+TEST(OrcTimestampDecodeContextTest, PhysicalTypeSelectsCivilOrInstantTimezone) {
+ OrcTimestampDecodeContext context(cctz::fixed_time_zone(cctz::seconds(8 * 3600)));
+ ASSERT_TRUE(cctz::load_time_zone("America/New_York", &context.reader_timezone));
+ orc::TimestampVectorBatch batch(1, *orc::getDefaultPool());
+ batch.numElements = 1;
+ batch.notNull[0] = 1;
+ batch.data[0] = 1;
+ batch.nanoseconds[0] = 0;
+ const DataTypeDateTimeV2 type(6);
+ for (const auto kind : {orc::TIMESTAMP, orc::TIMESTAMP_INSTANT}) {
+ auto file_type = orc::createPrimitiveType(kind);
+ OrcDecodedColumnView view {.file_type = file_type.get(),
+ .batch = &batch,
+ .rows = 1,
+ .timestamp_context = &context};
+ auto column = type.create_column();
+ auto status = type.get_serde()->read_column_from_orc(*column, view);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(type.to_string(*column, 0), kind == orc::TIMESTAMP
+ ? "1969-12-31 19:00:01.000000"
+ : "1970-01-01 08:00:01.000000");
+ }
+}
+
+TEST(OrcTimestampDecodeContextTest, NestedViewsKeepContextAcrossStripeTimezoneChanges) {
+ OrcTimestampDecodeContext context(cctz::fixed_time_zone(cctz::seconds(8 * 3600)));
+ auto schema = orc::createStructType();
+ schema->addStructField("plain", orc::createPrimitiveType(orc::TIMESTAMP));
+ schema->addStructField("instant", orc::createPrimitiveType(orc::TIMESTAMP_INSTANT));
+ orc::TimestampVectorBatch batch(1, *orc::getDefaultPool());
+ batch.numElements = 1;
+ batch.notNull[0] = 1;
+ batch.data[0] = 1;
+ batch.nanoseconds[0] = 123;
+ const std::vector<size_t> selected_rows {0};
+ OrcDecodedColumnView parent {.file_type = schema.get(),
+ .selected_type = schema.get(),
+ .rows = 1,
+ .timestamp_context = &context};
+ auto plain_view = orc_serde_utils::make_child_orc_view(
+ parent, schema->getSubtype(0), schema->getSubtype(0), &batch, 1, &selected_rows);
+ auto instant_view = orc_serde_utils::make_child_orc_view(
+ parent, schema->getSubtype(1), schema->getSubtype(1), &batch, 1, &selected_rows);
+ ASSERT_EQ(plain_view.timestamp_context, &context);
+ ASSERT_EQ(instant_view.timestamp_context, &context);
+
+ const DataTypeTimeStampNs plain_type;
+ const DataTypeDateTimeV2 instant_type(6);
+ for (const std::string zone : {"America/New_York", "UTC"}) {
+ ASSERT_TRUE(cctz::load_time_zone(zone, &context.reader_timezone));
+ auto plain_column = plain_type.create_column();
+ auto status = plain_type.get_serde()->read_column_from_orc(*plain_column, plain_view);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(
+ plain_type.to_string(*plain_column, 0),
+ zone == "UTC" ? "1970-01-01 00:00:01.000000123" : "1969-12-31 19:00:01.000000123");
+ auto instant_column = instant_type.create_column();
+ status = instant_type.get_serde()->read_column_from_orc(*instant_column, instant_view);
+ ASSERT_TRUE(status.ok()) << status;
+ // Changing a stripe's reader zone must never change an instant sibling's display zone.
+ EXPECT_EQ(instant_type.to_string(*instant_column, 0), "1970-01-01 08:00:01.000000");
+ }
+}
+
+} // namespace doris
diff --git a/be/test/exec/connector/jni_connector_test.cpp b/be/test/exec/connector/jni_connector_test.cpp
new file mode 100644
index 00000000000..4f164ec6674
--- /dev/null
+++ b/be/test/exec/connector/jni_connector_test.cpp
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/connector/jni_connector.h"
+
+#include <gtest/gtest.h>
+
+#include <array>
+#include <limits>
+
+#include "core/block/block.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_timestamp_ns.h"
+
+namespace doris {
+
+TEST(JniConnectorTest, TimestampNsPreservesSignedValuesAndBatchOffsets) {
+ auto type = make_nullable(std::make_shared<DataTypeTimeStampNs>());
+ EXPECT_EQ(JniConnector::get_jni_type(type), "timestamp_ns");
+ EXPECT_EQ(JniConnector::get_jni_type_with_different_string(type), "timestamp_ns");
+ Block block({ColumnWithTypeAndName(type->create_column(), type, "ts")});
+ std::array<uint8_t, 4> nulls {0, 0, 1, 0};
+ std::array<int64_t, 4> values {std::numeric_limits<int64_t>::min(), -1, 0,
+ std::numeric_limits<int64_t>::max()};
+ std::array<long, 3> meta {4, reinterpret_cast<long>(nulls.data()),
+ reinterpret_cast<long>(values.data())};
+ // Appending a second batch must advance by timestamp elements, not raw bytes.
+ for (int batch = 0; batch < 2; ++batch) {
+ ASSERT_TRUE(
+ JniConnector::fill_block(&block, {0}, reinterpret_cast<long>(meta.data())).ok());
+ }
+ const auto& nullable = assert_cast<const ColumnNullable&>(*block.get_by_position(0).column);
+ const auto& data =
+ assert_cast<const ColumnTimeStampNs&>(nullable.get_nested_column()).get_data();
+ ASSERT_EQ(data.size(), 8);
+ for (size_t i = 0; i < data.size(); ++i) {
+ EXPECT_EQ(data[i].epoch_nanos(), values[i % values.size()]);
+ EXPECT_EQ(nullable.get_null_map_data()[i], nulls[i % nulls.size()]);
+ }
+}
+
+} // namespace doris
diff --git a/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp b/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp
index f6909877350..6ebf168af0c 100644
--- a/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp
+++ b/be/test/exec/sink/writer/iceberg/partition_transformers_test.cpp
@@ -23,8 +23,11 @@
#include <tuple>
#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_timestamptz.h"
+#include "core/data_type/data_type_uuid.h"
#include "core/data_type/data_type_varbinary.h"
+#include "core/value/uuid_value.h"
#include "format/table/iceberg/partition_spec.h"
namespace doris {
@@ -146,17 +149,24 @@ TEST_F(PartitionTransformersTest, binary_truncate_counts_bytes_and_preserves_typ
TEST_F(PartitionTransformersTest, timestamp_transforms_use_utc_calendar_and_microseconds) {
for (const auto& type : std::vector<DataTypePtr> {std::make_shared<DataTypeDateTimeV2>(6),
- std::make_shared<DataTypeTimeStampTz>(6)}) {
+ std::make_shared<DataTypeTimeStampTz>(6),
+ std::make_shared<DataTypeTimeStampNs>()}) {
auto column = type->create_column();
// The first two UTC values represent the repeated New York 01:30 with different offsets.
const std::vector<std::array<int, 7>> fields = {{2021, 11, 7, 5, 30, 0, 123456},
{2021, 11, 7, 6, 30, 0, 123456},
{1969, 12, 31, 23, 59, 59, 999999}};
const std::vector<Int64> micros = {1636263000123456, 1636266600123456, -1};
+ size_t index = 0;
for (const auto& f : fields) {
DateV2Value<DateTimeV2ValueType> dt;
ASSERT_TRUE(dt.check_range_and_set_time(f[0], f[1], f[2], f[3], f[4], f[5], f[6]));
auto packed = dt.to_date_int_val();
+ if (type->get_primitive_type() == TYPE_TIMESTAMP_NS) {
+ // All sub-microsecond fractions must bucket using the floor microsecond.
+ packed = static_cast<uint64_t>(micros[index] * 1000 + 999);
+ }
+ ++index;
column->insert_data(reinterpret_cast<const char*>(&packed), sizeof(packed));
}
column->insert_default();
@@ -748,4 +758,23 @@ TEST_F(PartitionTransformersTest, DatePartitionsMatchYearZeroFileOrdinals) {
}
}
+TEST_F(PartitionTransformersTest, uuid_identity_and_bucket_preserve_logical_value) {
+ auto type = std::make_shared<DataTypeUUID>();
+ auto column = ColumnUUID::create();
+ UUIDValueType value;
+ ASSERT_TRUE(UUIDValue::from_string(value, "00112233-4455-6677-8899-aabbccddeeff"));
+ column->insert_value(value);
+ Block block({{std::move(column), type, "u"}});
+ iceberg::PartitionField field(1, 1000, "u_bucket", "bucket[17]");
+ auto transform = PartitionColumnTransforms::create(field, type);
+ auto result = transform->apply(block, 0);
+ const auto bytes = UUIDValue::to_big_endian(value);
+ EXPECT_EQ(assert_cast<const ColumnInt32&>(*result.column).get_data()[0],
+ (HashUtil::murmur_hash3_32(bytes.data(), bytes.size(), 0) & INT32_MAX) % 17);
+ IdentityPartitionColumnTransform identity(type);
+ EXPECT_EQ(
+ identity.get_partition_value(type, std::string("00112233-4455-6677-8899-aabbccddeeff")),
+ "00112233-4455-6677-8899-aabbccddeeff");
+}
+
} // namespace doris
diff --git a/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp b/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp
index 1bbbe5454af..0d70c865c7e 100644
--- a/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp
+++ b/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp
@@ -66,7 +66,7 @@ public:
TEST(VIcebergTableWriterTest, TimestampIdentityPreservesUtcCommitAndNull) {
std::vector<iceberg::NestedField> fields;
- fields.emplace_back(false, 1, "event_time", std::make_unique<iceberg::TimestampType>(true),
+ fields.emplace_back(false, 1, "event_time", std::make_unique<iceberg::TimestampTzType>(),
std::nullopt);
auto schema = std::make_shared<iceberg::Schema>(std::move(fields));
auto spec = iceberg::PartitionSpecParser::from_json(
diff --git a/be/test/format/table/iceberg/arrow_schema_util_test.cpp b/be/test/format/table/iceberg/arrow_schema_util_test.cpp
index 6610015ed38..37346e0462a 100644
--- a/be/test/format/table/iceberg/arrow_schema_util_test.cpp
+++ b/be/test/format/table/iceberg/arrow_schema_util_test.cpp
@@ -23,6 +23,7 @@
#include <arrow/status.h>
#include <arrow/type.h>
#include <arrow/util/key_value_metadata.h>
+#include <fmt/format.h>
#include <gtest/gtest.h>
#include <parquet/api/reader.h>
#include <parquet/arrow/writer.h>
@@ -124,6 +125,30 @@ TEST(ArrowSchemaUtilTest, IcebergParquetWriterPreservesUuidLogicalAnnotations) {
EXPECT_TRUE(metadata->key_value_metadata()->Contains("ARROW:schema"));
}
+TEST(ArrowSchemaUtilTest, TimestampKindsPreserveUnitAndTimezone) {
+ const char* names[] = {"timestamp", "timestamptz", "timestamp_ns", "timestamptz_ns"};
+ const arrow::TimeUnit::type units[] = {arrow::TimeUnit::MICRO, arrow::TimeUnit::MICRO,
+ arrow::TimeUnit::NANO, arrow::TimeUnit::NANO};
+ const char* zones[] = {"", "Asia/Shanghai", "", "Asia/Shanghai"};
+ for (int i = 0; i < 4; ++i) {
+ SCOPED_TRACE(names[i]);
+ const auto json = fmt::format(R"({{"type":"struct","fields":[
+ {{"id":1,"name":"ts","required":false,"type":"{0}"}},
+ {{"id":2,"name":"nested","required":false,"type":{{"type":"list",
+ "element-id":3,"element-required":false,"element":"{0}"}}}}]}})",
+ names[i]);
+ auto schema = SchemaParser::from_json(json);
+ IcebergArrowSchemaConvertor converter(*schema, "Asia/Shanghai", json);
+ std::shared_ptr<arrow::Schema> arrow_schema;
+ ASSERT_TRUE(converter.get_arrow_schema(&arrow_schema).ok());
+ const auto expected = arrow::timestamp(units[i], zones[i]);
+ EXPECT_TRUE(arrow_schema->field(0)->type()->Equals(expected));
+ auto list_type = std::static_pointer_cast<arrow::ListType>(arrow_schema->field(1)->type());
+ EXPECT_TRUE(list_type->value_type()->Equals(expected));
+ EXPECT_EQ(schema->columns()[0].field_type()->to_string(), names[i]);
+ }
+}
+
const std::string_view pfid = "PARQUET:field_id";
TEST(ArrowSchemaUtilTest, test_simple_field) {
diff --git a/be/test/format/table/iceberg/type_test.cpp b/be/test/format/table/iceberg/type_test.cpp
index 6c24037f11a..8ede6af24b7 100644
--- a/be/test/format/table/iceberg/type_test.cpp
+++ b/be/test/format/table/iceberg/type_test.cpp
@@ -65,13 +65,34 @@ TEST(StringTypeTest, to_string) {
}
TEST(TimestampTypeTest, to_string) {
- TimestampType timestamp_type_with_utc(true);
+ TimestampTzType timestamp_type_with_utc;
EXPECT_EQ(timestamp_type_with_utc.to_string(), "timestamptz");
- EXPECT_EQ(timestamp_type_with_utc.type_id(), TypeID::TIMESTAMP);
+ EXPECT_EQ(timestamp_type_with_utc.type_id(), TypeID::TIMESTAMPTZ);
- TimestampType timestamp_type_without_utc(false);
+ TimestampType timestamp_type_without_utc;
EXPECT_EQ(timestamp_type_without_utc.to_string(), "timestamp");
EXPECT_EQ(timestamp_type_without_utc.type_id(), TypeID::TIMESTAMP);
+
+ TimestampNsType timestamp_ns;
+ EXPECT_EQ(timestamp_ns.to_string(), "timestamp_ns");
+ EXPECT_EQ(timestamp_ns.type_id(), TypeID::TIMESTAMP_NS);
+
+ TimestampTzNsType timestamptz_ns;
+ EXPECT_EQ(timestamptz_ns.to_string(), "timestamptz_ns");
+ EXPECT_EQ(timestamptz_ns.type_id(), TypeID::TIMESTAMPTZ_NS);
+}
+
+TEST(TimestampTypeTest, ParsedTimestampKindsHaveDistinctIds) {
+ const char* names[] = {"timestamp", "timestamptz", "timestamp_ns", "timestamptz_ns"};
+ // Type-based dispatch must preserve both the timestamp unit and timezone semantics.
+ for (int i = 0; i < 4; ++i) {
+ auto type = Types::from_primitive_string(names[i]);
+ EXPECT_EQ(type->to_string(), names[i]);
+ for (int j = 0; j < i; ++j) {
+ EXPECT_NE(type->type_id(), Types::from_primitive_string(names[j])->type_id())
+ << names[i] << " and " << names[j];
+ }
+ }
}
TEST(TimeTypeTest, to_string) {
@@ -124,8 +145,10 @@ TEST(PrimitiveTypesTest, from_primitive_string) {
EXPECT_EQ(Types::from_primitive_string("double")->type_id(), TypeID::DOUBLE);
EXPECT_EQ(Types::from_primitive_string("date")->type_id(), TypeID::DATE);
EXPECT_EQ(Types::from_primitive_string("time")->type_id(), TypeID::TIME);
- EXPECT_EQ(Types::from_primitive_string("timestamptz")->type_id(), TypeID::TIMESTAMP);
+ EXPECT_EQ(Types::from_primitive_string("timestamptz")->type_id(), TypeID::TIMESTAMPTZ);
EXPECT_EQ(Types::from_primitive_string("timestamp")->type_id(), TypeID::TIMESTAMP);
+ EXPECT_EQ(Types::from_primitive_string("timestamp_ns")->type_id(), TypeID::TIMESTAMP_NS);
+ EXPECT_EQ(Types::from_primitive_string("timestamptz_ns")->type_id(), TypeID::TIMESTAMPTZ_NS);
EXPECT_EQ(Types::from_primitive_string("string")->type_id(), TypeID::STRING);
EXPECT_EQ(Types::from_primitive_string("uuid")->type_id(), TypeID::UUID);
EXPECT_EQ(Types::from_primitive_string("binary")->type_id(), TypeID::BINARY);
diff --git a/be/test/format/transformer/vorc_transformer_test.cpp b/be/test/format/transformer/vorc_transformer_test.cpp
index 1e223e899cc..f0808e464c9 100644
--- a/be/test/format/transformer/vorc_transformer_test.cpp
+++ b/be/test/format/transformer/vorc_transformer_test.cpp
@@ -36,6 +36,7 @@
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_timestamptz.h"
#include "core/data_type/data_type_uuid.h"
#include "core/data_type/data_type_varbinary.h"
@@ -54,6 +55,8 @@ namespace doris {
class VOrcTransformerTest : public testing::Test {
protected:
void SetUp() override {
+ // Writers use the startup timezone cache; each test must work without another suite initializing it.
+ TimezoneUtils::load_timezones_to_cache();
_file_path = "./vorc_transformer_" + UniqueId::gen_uid().to_string() + ".orc";
_fs = io::global_local_filesystem();
}
@@ -89,6 +92,51 @@ TEST_F(VOrcTransformerTest, RejectsLossyPreEpochFractions) {
}
}
+TEST_F(VOrcTransformerTest, NanosecondBoundsUseUtcWallClockCarrier) {
+ auto type = std::make_shared<DataTypeTimeStampNs>();
+ auto exprs = MockSlotRef::create_mock_contexts(DataTypes {type});
+ const std::string json =
+ R"({"type":"struct","fields":[{"id":1,"name":"ts","required":true,"type":"timestamp_ns"}]})";
+ auto schema = iceberg::SchemaParser::from_json(json);
+ io::FileWriterPtr file_writer;
+ ASSERT_TRUE(_fs->create_file(_file_path, &file_writer).ok());
+ RuntimeState state;
+ state.set_timezone("Asia/Shanghai");
+ VOrcTransformer writer(&state, file_writer.get(), exprs, "", {"ts"}, false,
+ TFileCompressType::PLAIN, schema.get(), _fs);
+ ASSERT_TRUE(writer.open().ok());
+ const std::vector<int64_t> nanos = {
+ std::numeric_limits<int64_t>::min(), -1000000001, 0, 1, 123456789,
+ std::numeric_limits<int64_t>::max()};
+ auto column = type->create_column();
+ for (auto value : nanos) {
+ column->insert(Field::create_field<TYPE_TIMESTAMP_NS>(TimeStampNsValue(value)));
+ }
+ const auto write_status = writer.write(Block({{std::move(column), type, "ts"}}));
+ ASSERT_TRUE(write_status.ok()) << write_status;
+ ASSERT_TRUE(writer.close().ok());
+ TIcebergColumnStats stats;
+ ASSERT_TRUE(writer.collect_file_statistics_after_close(&stats).ok());
+ EXPECT_FALSE(stats.lower_bounds.contains(1));
+ EXPECT_FALSE(stats.upper_bounds.contains(1));
+ auto reader = orc::createReader(orc::readLocalFile(_file_path), orc::ReaderOptions());
+ EXPECT_TRUE(orc_serde_utils::has_nanosecond_timestamp(reader->getType()));
+ orc::RowReaderOptions options;
+ options.setTimezoneName("GMT");
+ auto rows = reader->createRowReader(options);
+ auto batch = rows->createRowBatch(nanos.size());
+ ASSERT_TRUE(rows->next(*batch));
+ const auto& values = assert_cast<const orc::TimestampVectorBatch&>(
+ *assert_cast<const orc::StructVectorBatch&>(*batch).fields[0]);
+ ASSERT_EQ(nanos.size(), batch->numElements);
+ for (size_t row = 0; row < nanos.size(); ++row) {
+ const __int128 actual =
+ static_cast<__int128>(values.data[row]) * 1000000000 + values.nanoseconds[row];
+ EXPECT_EQ(static_cast<__int128>(nanos[row]), actual);
+ }
+}
+
+// NOLINTNEXTLINE(readability-function-cognitive-complexity): assertions cover the timezone/type matrix.
TEST_F(VOrcTransformerTest, TimestampBoundsRetainMicroseconds) {
TimezoneUtils::load_timezones_to_cache();
// UTC aliases must share ORC's UTC encoding, including names without a zoneinfo file.
@@ -454,6 +502,48 @@ TEST_F(VOrcTransformerTest, PreservesVarbinaryUuidCarrierBeforeOrcWrite) {
EXPECT_EQ(0, std::memcmp(uuid_batch.data[0], expected_uuid.data(), expected_uuid.size()));
}
+TEST_F(VOrcTransformerTest, PreservesNativeUuidCarrierBeforeIcebergOrcWrite) {
+ const std::string schema_json = R"({
+ "type": "struct",
+ "fields": [
+ {"id": 1, "name": "uuid_col", "required": true, "type": "uuid"}
+ ]
+ })";
+ std::unique_ptr<iceberg::Schema> schema = iceberg::SchemaParser::from_json(schema_json);
+ auto uuid_type = std::make_shared<DataTypeUUID>();
+ VExprContextSPtrs output_exprs = MockSlotRef::create_mock_contexts(DataTypes {uuid_type});
+
+ io::FileWriterPtr file_writer;
+ ASSERT_TRUE(_fs->create_file(_file_path, &file_writer).ok());
+ RuntimeState state;
+ state.set_timezone("UTC");
+ VOrcTransformer transformer(&state, file_writer.get(), output_exprs, "", {"uuid_col"}, false,
+ TFileCompressType::PLAIN, schema.get(), _fs);
+ ASSERT_TRUE(transformer.open().ok());
+
+ const std::array<uint8_t, 16> expected_uuid = {0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77,
+ 0x88, 0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff};
+ auto uuid_column = ColumnUUID::create();
+ uuid_column->get_data().push_back(UUIDValue::from_big_endian(expected_uuid.data()));
+ Block block;
+ block.insert({std::move(uuid_column), uuid_type, "uuid_col"});
+ ASSERT_TRUE(transformer.write(block).ok());
+ ASSERT_TRUE(transformer.close().ok());
+
+ io::FileReaderSPtr file_reader;
+ ASSERT_TRUE(_fs->open_file(_file_path, &file_reader).ok());
+ auto input_stream = std::make_unique<ORCFileInputStream>(
+ _file_path, file_reader, nullptr, nullptr, 8L * 1024L * 1024L, 1L * 1024L * 1024L);
+ auto reader = orc::createReader(std::move(input_stream), orc::ReaderOptions());
+ auto row_reader = reader->createRowReader();
+ auto row_batch = row_reader->createRowBatch(1);
+ ASSERT_TRUE(row_reader->next(*row_batch));
+ const auto& root = assert_cast<const orc::StructVectorBatch&>(*row_batch);
+ const auto& uuid_batch = assert_cast<const orc::StringVectorBatch&>(*root.fields[0]);
+ EXPECT_EQ(uuid_batch.length[0], expected_uuid.size());
+ EXPECT_EQ(0, std::memcmp(uuid_batch.data[0], expected_uuid.data(), expected_uuid.size()));
+}
+
TEST_F(VOrcTransformerTest, RejectsInvalidLegacyUuidAndFixedValues) {
const std::string schema_json = R"({
"type": "struct",
diff --git a/be/test/format/transformer/vparquet_writer_test.cpp b/be/test/format/transformer/vparquet_writer_test.cpp
index 279d0e664d0..cfdf4069aae 100644
--- a/be/test/format/transformer/vparquet_writer_test.cpp
+++ b/be/test/format/transformer/vparquet_writer_test.cpp
@@ -34,7 +34,10 @@
#include "core/data_type/data_type_array.h"
#include "core/data_type/data_type_factory.hpp"
#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_variant_v2.h"
+#include "exec/sink/writer/iceberg/viceberg_partition_writer.h"
#include "exec/sink/writer/vhive_partition_writer.h"
#include "exprs/function/parse/variant_string_parse.h"
#include "format/table/iceberg/schema_parser.h"
@@ -384,4 +387,53 @@ TEST_F(VParquetWriterTest, HiveInt96HonorsCatalogTimezoneContract) {
}
}
+TEST_F(VParquetWriterTest, IcebergNanosecondsRoundTripWithoutCoercion) {
+ const auto type = make_nullable(std::make_shared<DataTypeTimeStampNs>());
+ auto exprs = MockSlotRef::create_mock_contexts(DataTypes {type});
+ const std::string json =
+ R"({"type":"struct","fields":[{"id":1,"name":"ts","required":false,"type":"timestamp_ns"}]})";
+ auto schema = iceberg::SchemaParser::from_json(json);
+ const std::map<std::string, std::string> hadoop_conf;
+ TDataSink sink;
+ RuntimeState state;
+ state.set_timezone("Asia/Shanghai");
+ const auto name = "iceberg_ns_" + UniqueId::gen_uid().to_string();
+ _file_path = "./" + name + "-0.parquet";
+ VIcebergPartitionWriter writer(sink, {}, exprs, *schema, &json, {"ts"},
+ {.write_path = ".",
+ .original_write_path = ".",
+ .target_path = ".",
+ .file_type = TFileType::FILE_LOCAL,
+ .broker_addresses = {}},
+ name, 0, TFileFormatType::FORMAT_PARQUET,
+ TFileCompressType::PLAIN, hadoop_conf);
+ ASSERT_TRUE(writer.open(&state, nullptr, nullptr).ok());
+ const std::vector<int64_t> nanos = {std::numeric_limits<int64_t>::min(), -1, 0, 1, 123456789,
+ std::numeric_limits<int64_t>::max()};
+ auto column = type->create_column();
+ for (auto value : nanos) {
+ column->insert(Field::create_field<TYPE_TIMESTAMP_NS>(TimeStampNsValue(value)));
+ }
+ column->insert_default();
+ Block block({{std::move(column), type, "ts"}});
+ auto status = writer.write(block);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_TRUE(writer.close(Status::OK()).ok());
+ auto input = arrow::io::ReadableFile::Open(_file_path);
+ ASSERT_TRUE(input.ok()) << input.status();
+ auto reader = ::parquet::arrow::OpenFile(*input, arrow::default_memory_pool());
+ ASSERT_TRUE(reader.ok()) << reader.status();
+ auto table = (*reader)->ReadTable();
+ ASSERT_TRUE(table.ok()) << table.status();
+ const auto& values = assert_cast<const arrow::TimestampArray&>(*(*table)->column(0)->chunk(0));
+ const auto& timestamp_type = assert_cast<const arrow::TimestampType&>(*values.type());
+ EXPECT_EQ(arrow::TimeUnit::NANO, timestamp_type.unit());
+ EXPECT_TRUE(timestamp_type.timezone().empty());
+ ASSERT_EQ(nanos.size() + 1, values.length());
+ for (size_t row = 0; row < nanos.size(); ++row) {
+ EXPECT_EQ(nanos[row], values.Value(row));
+ }
+ EXPECT_TRUE(values.IsNull(nanos.size()));
+}
+
} // namespace doris
diff --git a/be/test/format_v2/expr/evolution_cast_test.cpp b/be/test/format_v2/expr/evolution_cast_test.cpp
new file mode 100644
index 00000000000..e9cd32d6562
--- /dev/null
+++ b/be/test/format_v2/expr/evolution_cast_test.cpp
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/expr/evolution_cast.h"
+
+#include <gtest/gtest.h>
+
+#include "core/block/block.h"
+#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_timestamp_ns.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+#include "testutil/column_helper.h"
+#include "testutil/mock/mock_runtime_state.h"
+
+namespace doris::format {
+
+class EvolutionCastTest : public testing::Test {
+protected:
+ static DateV2Value<DateV2ValueType> date(int year, int month, int day) {
+ DateV2Value<DateV2ValueType> value;
+ value.unchecked_set_time(year, month, day, 0, 0, 0, 0);
+ return value;
+ }
+
+ Status execute(const VExprSPtr& expr, Block* block, int* result) {
+ auto context = VExprContext::create_shared(expr);
+ auto status = context->prepare(&state, RowDescriptor());
+ if (status.ok()) {
+ status = context->open(&state);
+ }
+ if (status.ok()) {
+ status = context->execute(block, result);
+ }
+ context->close();
+ return status;
+ }
+
+ DataTypePtr source_type = make_nullable(std::make_shared<DataTypeDateV2>());
+ DataTypePtr target_type = make_nullable(std::make_shared<DataTypeTimeStampNs>());
+ MockRuntimeState state;
+};
+
+TEST_F(EvolutionCastTest, OverflowIsIndependentOfSessionAndSurvivesClone) {
+ for (bool strict : {false, true}) {
+ state.set_enable_strict_cast(strict);
+ for (auto value :
+ {date(1000, 1, 1), date(1677, 9, 21), date(2262, 4, 12), date(9999, 12, 31)}) {
+ for (bool clone : {false, true}) {
+ SCOPED_TRACE(testing::Message() << "strict=" << strict << " clone=" << clone);
+ VExprSPtr expr = EvolutionCast::create_shared(target_type);
+ expr->add_child(VSlotRef::create_shared(0, 0, -1, source_type, "source"));
+ if (clone) {
+ VExprSPtr cloned;
+ ASSERT_TRUE(expr->deep_clone(&cloned).ok());
+ expr = std::move(cloned);
+ }
+ Block block;
+ block.insert({ColumnHelper::create_nullable_column<DataTypeDateV2>({value}, {0}),
+ source_type, "source"});
+ int result = -1;
+ auto status = execute(expr, &block, &result);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("TIMESTAMP_NS overflow"), std::string::npos)
+ << status;
+ }
+ }
+ }
+}
+
+TEST_F(EvolutionCastTest, ValidDatesAndNullRemainUnchanged) {
+ state.set_enable_strict_cast(false);
+ auto expr = EvolutionCast::create_shared(target_type);
+ expr->add_child(VSlotRef::create_shared(0, 0, -1, source_type, "source"));
+ Block block;
+ // A NULL payload may be outside the target range and must not trigger overflow.
+ block.insert({ColumnHelper::create_nullable_column<DataTypeDateV2>(
+ {date(1677, 9, 22), date(2262, 4, 11), date(1000, 1, 1)}, {0, 0, 1}),
+ source_type, "source"});
+ int result = -1;
+ ASSERT_TRUE(execute(expr, &block, &result).ok());
+ const auto& column = *block.get_by_position(result).column;
+ EXPECT_EQ(target_type->to_string(column, 0), "1677-09-22 00:00:00.000000000");
+ EXPECT_EQ(target_type->to_string(column, 1), "2262-04-11 00:00:00.000000000");
+ EXPECT_TRUE(column.is_null_at(2));
+}
+
+TEST_F(EvolutionCastTest, ConstantOverflowFailsDuringOpen) {
+ state.set_enable_strict_cast(false);
+ auto expr = EvolutionCast::create_shared(target_type);
+ expr->add_child(VLiteral::create_shared(std::make_shared<DataTypeDateV2>(),
+ Field::create_field<TYPE_DATEV2>(date(1000, 1, 1))));
+ auto context = VExprContext::create_shared(expr);
+ ASSERT_TRUE(context->prepare(&state, RowDescriptor()).ok());
+ auto status = context->open(&state);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("TIMESTAMP_NS overflow"), std::string::npos) << status;
+ context->close();
+}
+
+TEST_F(EvolutionCastTest, OrdinaryCastPreservesSessionBehavior) {
+ for (bool strict : {false, true}) {
+ state.set_enable_strict_cast(strict);
+ auto expr = Cast::create_shared(target_type);
+ expr->add_child(VSlotRef::create_shared(0, 0, -1, source_type, "source"));
+ Block block;
+ block.insert({ColumnHelper::create_nullable_column<DataTypeDateV2>({date(1000, 1, 1)}, {0}),
+ source_type, "source"});
+ int result = -1;
+ auto status = execute(expr, &block, &result);
+ if (strict) {
+ EXPECT_FALSE(status.ok());
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(block.get_by_position(result).column->is_null_at(0));
+ }
+ }
+}
+
+TEST_F(EvolutionCastTest, RuntimeContextCloneRetainsOverflowCheck) {
+ state.set_enable_strict_cast(false);
+ auto expr = EvolutionCast::create_shared(target_type);
+ expr->add_child(VSlotRef::create_shared(0, 0, -1, source_type, "source"));
+ auto context = VExprContext::create_shared(expr);
+ ASSERT_TRUE(context->prepare(&state, RowDescriptor()).ok());
+ ASSERT_TRUE(context->open(&state).ok());
+ VExprContextSPtr cloned;
+ ASSERT_TRUE(context->clone(&state, cloned).ok());
+ Block block;
+ block.insert({ColumnHelper::create_nullable_column<DataTypeDateV2>({date(1000, 1, 1)}, {0}),
+ source_type, "source"});
+ int result = -1;
+ auto status = cloned->execute(&block, &result);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("TIMESTAMP_NS overflow"), std::string::npos) << status;
+ cloned->close();
+ context->close();
+}
+
+TEST_F(EvolutionCastTest, EmptyInputRemainsEmpty) {
+ state.set_enable_strict_cast(false);
+ auto expr = EvolutionCast::create_shared(target_type);
+ expr->add_child(VSlotRef::create_shared(0, 0, -1, source_type, "source"));
+ Block block;
+ block.insert(
+ {ColumnHelper::create_nullable_column<DataTypeDateV2>({}, {}), source_type, "source"});
+ int result = -1;
+ ASSERT_TRUE(execute(expr, &block, &result).ok());
+ EXPECT_EQ(block.get_by_position(result).column->size(), 0);
+}
+
+TEST_F(EvolutionCastTest, NumericPromotionUsesExistingConversion) {
+ auto expr = EvolutionCast::create_shared(std::make_shared<DataTypeInt64>());
+ expr->add_child(VSlotRef::create_shared(0, 0, -1, std::make_shared<DataTypeInt32>(), "source"));
+ Block block;
+ block.insert(ColumnHelper::create_column_with_name<DataTypeInt32>({1, -2, 3}));
+ int result = -1;
+ ASSERT_TRUE(execute(expr, &block, &result).ok());
+ const auto& column = assert_cast<const ColumnInt64&>(*block.get_by_position(result).column);
+ EXPECT_EQ(column.get_data()[0], 1);
+ EXPECT_EQ(column.get_data()[1], -2);
+ EXPECT_EQ(column.get_data()[2], 3);
+}
+
+} // namespace doris::format
diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp
index 0953d5f2525..2e9e879cf2c 100644
--- a/be/test/format_v2/orc/orc_reader_test.cpp
+++ b/be/test/format_v2/orc/orc_reader_test.cpp
@@ -51,6 +51,7 @@
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
#include "core/column/column_struct.h"
+#include "core/column/column_varbinary.h"
#include "core/column/column_vector.h"
#include "core/data_type/data_type_array.h"
#include "core/data_type/data_type_date.h"
@@ -62,6 +63,7 @@
#include "core/data_type/data_type_timestamptz.h"
#include "core/data_type/data_type_varbinary.h"
#include "core/data_type/primitive_type.h"
+#include "core/data_type_serde/orc_timestamp_decode_context.h"
#include "core/value/timestamptz_value.h"
#include "exprs/create_predicate_function.h"
#include "exprs/runtime_filter_expr.h"
@@ -76,6 +78,7 @@
#include "format_v2/expr/delete_predicate.h"
#include "format_v2/file_reader.h"
#include "format_v2/parquet/parquet_profile.h"
+#include "format_v2/schema_reader.h"
#include "gen_cpp/Types_types.h"
#include "io/fs/buffered_reader.h"
#include "io/io_common.h"
@@ -1691,7 +1694,8 @@ public:
size_t count, ColumnPtr& result_column) const override {
const auto& nullable_column =
assert_cast<const ColumnNullable&>(*block->get_by_position(_column_id).column);
- const auto& input = assert_cast<const ColumnString&>(nullable_column.get_nested_column());
+ // Residual test comparisons apply to raw bytes in both string and binary columns.
+ const auto& input = nullable_column.get_nested_column();
auto result = ColumnUInt8::create();
auto& result_data = result->get_data();
result_data.resize(count);
@@ -4762,7 +4766,7 @@ TEST_F(NewOrcReaderTest, UuidBinaryRequiresLogicalTypeAttribute) {
auto reader = create_reader();
auto binary = ::orc::createPrimitiveType(::orc::BINARY);
EXPECT_EQ(remove_nullable(reader->_convert_to_doris_type(*binary))->get_primitive_type(),
- TYPE_STRING);
+ TYPE_VARBINARY);
binary->setAttribute("doris.logical_type", "uuid");
EXPECT_EQ(remove_nullable(reader->_convert_to_doris_type(*binary))->get_primitive_type(),
TYPE_UUID);
@@ -8112,7 +8116,7 @@ TEST_F(NewOrcReaderTest, SargBinaryVarbinaryLiteralConjunctPrunesRowGroups) {
std::vector<format::ColumnDefinition> schema;
ASSERT_TRUE(reader->get_schema(&schema).ok());
ASSERT_EQ(schema.size(), 2);
- EXPECT_EQ(remove_nullable(schema[0].type)->get_primitive_type(), TYPE_STRING);
+ EXPECT_EQ(remove_nullable(schema[0].type)->get_primitive_type(), TYPE_VARBINARY);
auto request = std::make_shared<format::FileScanRequest>();
request->predicate_columns = {field_projection(0)};
@@ -8170,7 +8174,7 @@ TEST_F(NewOrcReaderTest, SargExceptionsFallBackToResidualScan) {
if (add_real_residual_filter) {
request->conjuncts.push_back(
VExprContext::create_shared(std::make_shared<NullableStringEqualsExpr>(
- 0, std::make_shared<DataTypeString>(), "aaa_1050", "value")));
+ 0, std::make_shared<DataTypeVarbinary>(), "aaa_1050", "value")));
}
*open_status = reader->open(request);
if (!open_status->ok()) {
@@ -8188,7 +8192,7 @@ TEST_F(NewOrcReaderTest, SargExceptionsFallBackToResidualScan) {
const auto& values_nullable =
assert_cast<const ColumnNullable&>(*block.get_by_position(0).column);
const auto& values =
- assert_cast<const ColumnString&>(values_nullable.get_nested_column());
+ assert_cast<const ColumnVarbinary&>(values_nullable.get_nested_column());
for (size_t row = 0; row < rows; ++row) {
ASSERT_FALSE(values_nullable.is_null_at(row));
result_values->push_back(values.get_data_at(row).to_string());
@@ -8281,7 +8285,7 @@ TEST_F(NewOrcReaderTest, SargBinaryCastFromVarbinaryToStringConjunctPrunesRowGro
std::vector<format::ColumnDefinition> schema;
ASSERT_TRUE(reader->get_schema(&schema).ok());
ASSERT_EQ(schema.size(), 2);
- EXPECT_EQ(remove_nullable(schema[0].type)->get_primitive_type(), TYPE_STRING);
+ EXPECT_EQ(remove_nullable(schema[0].type)->get_primitive_type(), TYPE_VARBINARY);
auto request = std::make_shared<format::FileScanRequest>();
request->predicate_columns = {field_projection(0)};
@@ -10454,6 +10458,34 @@ TEST_F(NewOrcReaderTest, ReadsOnlyRequestedAbsoluteFileRows) {
EXPECT_EQ(ids, std::vector<int32_t>({1, 3, 5}));
}
+TEST_F(NewOrcReaderTest, NestedBinarySchemaRetainsBinarySemantics) {
+ auto type = ::orc::Type::buildTypeFromString(
+ "struct<b:binary,s:string,a:array<binary>,m:map<string,binary>,r:struct<b:binary>>");
+ MemoryOutputStream stream(1024);
+ ::orc::WriterOptions options;
+ auto writer = ::orc::createWriter(*type, &stream, options);
+ writer->close();
+ const auto path = (_test_dir / "nested_binary.orc").string();
+ std::ofstream out(path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ out.close();
+ auto reader = create_reader_for_path(path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ ASSERT_EQ(schema.size(), 5);
+ EXPECT_EQ(schema[0].type->get_primitive_type(), TYPE_VARBINARY);
+ EXPECT_EQ(schema[1].type->get_primitive_type(), TYPE_STRING);
+ const auto& array = assert_cast<const DataTypeArray&>(*remove_nullable(schema[2].type));
+ EXPECT_EQ(array.get_nested_type()->get_primitive_type(), TYPE_VARBINARY);
+ const auto& map = assert_cast<const DataTypeMap&>(*remove_nullable(schema[3].type));
+ EXPECT_EQ(map.get_key_type()->get_primitive_type(), TYPE_STRING);
+ EXPECT_EQ(map.get_value_type()->get_primitive_type(), TYPE_VARBINARY);
+ const auto& structure = assert_cast<const DataTypeStruct&>(*remove_nullable(schema[4].type));
+ EXPECT_EQ(structure.get_elements()[0]->get_primitive_type(), TYPE_VARBINARY);
+}
+
TEST_F(NewOrcReaderTest, ReadPrimitiveTypesWithNulls) {
const auto primitive_file_path = (_test_dir / "primitive.orc").string();
write_primitive_orc_file(primitive_file_path);
@@ -10473,7 +10505,7 @@ TEST_F(NewOrcReaderTest, ReadPrimitiveTypesWithNulls) {
EXPECT_EQ(remove_nullable(schema[5].type)->get_primitive_type(), TYPE_FLOAT);
EXPECT_EQ(remove_nullable(schema[6].type)->get_primitive_type(), TYPE_DOUBLE);
EXPECT_EQ(remove_nullable(schema[7].type)->get_primitive_type(), TYPE_STRING);
- EXPECT_EQ(remove_nullable(schema[8].type)->get_primitive_type(), TYPE_STRING);
+ ASSERT_EQ(remove_nullable(schema[8].type)->get_primitive_type(), TYPE_VARBINARY);
EXPECT_EQ(remove_nullable(schema[9].type)->get_primitive_type(), TYPE_VARCHAR);
EXPECT_EQ(remove_nullable(schema[10].type)->get_primitive_type(), TYPE_CHAR);
EXPECT_EQ(remove_nullable(schema[11].type)->get_primitive_type(), TYPE_DATEV2);
@@ -10528,7 +10560,7 @@ TEST_F(NewOrcReaderTest, ReadPrimitiveTypesWithNulls) {
const auto& string_values = assert_cast<const ColumnString&>(
assert_cast<const ColumnNullable&>(*block.get_by_position(7).column)
.get_nested_column());
- const auto& binary_values = assert_cast<const ColumnString&>(
+ const auto& binary_values = assert_cast<const ColumnVarbinary&>(
assert_cast<const ColumnNullable&>(*block.get_by_position(8).column)
.get_nested_column());
const auto& varchar_values = assert_cast<const ColumnString&>(
@@ -10616,6 +10648,366 @@ TEST_F(NewOrcReaderTest, ReadTimestampNanosecondsRoundsToMicroseconds) {
}
}
+TEST_F(NewOrcReaderTest, NanosecondFilesPreserveNegativeFractionsAcrossWriterTimezones) {
+ for (const auto& [writer_zone, offset] : std::vector<std::pair<std::string, int64_t>> {
+ {"GMT", 0}, {"Asia/Shanghai", 28800}, {"America/New_York", -18000}}) {
+ SCOPED_TRACE(writer_zone);
+ const auto file_path = (_test_dir / "timestamp_ns_writer_zone.orc").string();
+ auto type = ::orc::createStructType();
+ auto ns_type = ::orc::createPrimitiveType(::orc::TIMESTAMP);
+ ns_type->setAttribute("iceberg.timestamp-unit", "NANOS");
+ type->addStructField("ns", std::move(ns_type));
+ type->addStructField("plain", ::orc::createPrimitiveType(::orc::TIMESTAMP));
+ MemoryOutputStream stream(1024 * 1024);
+ ::orc::WriterOptions options;
+ options.setTimezoneName(writer_zone);
+ options.setCompression(::orc::CompressionKind_NONE);
+ auto writer = ::orc::createWriter(*type, &stream, options);
+ constexpr std::array<int64_t, 6> fractions {0, 1, 999999, 1000000, 500000123, 999999999};
+ constexpr size_t row_count = 3 * fractions.size();
+ auto batch = writer->createRowBatch(row_count);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ root.numElements = row_count;
+ for (auto* field : root.fields) {
+ auto& timestamps = dynamic_cast<::orc::TimestampVectorBatch&>(*field);
+ timestamps.numElements = row_count;
+ for (size_t row = 0; row < row_count; ++row) {
+ // Use the external writer's epoch carrier, not Doris's UTC writer convention.
+ timestamps.data[row] =
+ (static_cast<int64_t>(row / fractions.size()) - 1) * 2 - offset;
+ timestamps.nanoseconds[row] = fractions[row % fractions.size()];
+ }
+ }
+ writer->add(*batch);
+ writer->close();
+ {
+ std::ofstream out(file_path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ }
+ for (bool sparse : {false, true}) {
+ auto reader = create_reader_for_path(file_path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ state.set_timezone("+08:00");
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ Block block = build_file_block(schema);
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(0), field_projection(1)};
+ std::vector<int64_t> source_rows;
+ for (size_t row = 0; row < row_count; row += sparse ? 2 : 1) {
+ source_rows.push_back(static_cast<int64_t>(row));
+ }
+ if (sparse) {
+ request->row_ids = source_rows;
+ }
+ ASSERT_TRUE(reader->open(request).ok());
+ const std::array<std::string, 3> seconds {"1969-12-31 23:59:58", "1970-01-01 00:00:00",
+ "1970-01-01 00:00:02"};
+ const std::array<std::string, 3> carried_seconds {
+ "1969-12-31 23:59:59", "1970-01-01 00:00:01", "1970-01-01 00:00:03"};
+ const std::array<std::string, 6> micros {"000000", "000000", "001000",
+ "001000", "500000", "000000"};
+ bool eof = false;
+ size_t output_row = 0;
+ while (!eof) {
+ size_t rows = 0;
+ ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
+ for (size_t row = 0; row < rows; ++row) {
+ ASSERT_LT(output_row, source_rows.size());
+ const size_t source = static_cast<size_t>(source_rows[output_row++]);
+ const size_t second_index = source / fractions.size();
+ const size_t fraction_index = source % fractions.size();
+ EXPECT_EQ(schema[0].type->to_string(*block.get_by_position(0).column, row),
+ fmt::format("{}.{:09}", seconds[second_index],
+ fractions[fraction_index]));
+ const auto& whole = fraction_index == 5 ? carried_seconds : seconds;
+ EXPECT_EQ(schema[1].type->to_string(*block.get_by_position(1).column, row),
+ whole[second_index] + "." + micros[fraction_index]);
+ }
+ }
+ EXPECT_EQ(output_row, source_rows.size());
+ }
+ }
+}
+
+TEST_F(NewOrcReaderTest, NanosStripeFootersAreLazyAndRowReadersAreAmortized) {
+ const auto file = (_test_dir / "many_nanos_stripes.orc").string();
+ auto type = ::orc::createStructType();
+ auto ns = ::orc::createPrimitiveType(::orc::TIMESTAMP);
+ ns->setAttribute("iceberg.timestamp-unit", "NANOS");
+ type->addStructField("ns", std::move(ns));
+ MemoryOutputStream stream(1024 * 1024);
+ ::orc::WriterOptions options;
+ options.setTimezoneName("UTC");
+ options.setStripeSize(0);
+ auto writer = ::orc::createWriter(*type, &stream, options);
+ constexpr size_t STRIPES = 64;
+ for (size_t i = 0; i < STRIPES; ++i) {
+ auto batch = writer->createRowBatch(1);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ root.numElements = 1;
+ auto& ts = dynamic_cast<::orc::TimestampVectorBatch&>(*root.fields[0]);
+ ts.numElements = 1;
+ ts.data[0] = 0;
+ ts.nanoseconds[0] = static_cast<int64_t>(i);
+ writer->add(*batch);
+ }
+ writer->close();
+ {
+ std::ofstream out(file, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ }
+ ASSERT_EQ(get_orc_stripe_count(file), STRIPES);
+ RuntimeProfile profile("many_stripes");
+ auto reader = create_reader_for_path(file, &profile);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(0)};
+ ASSERT_TRUE(reader->open(request).ok());
+ ASSERT_EQ(profile.get_counter("OrcTimestampStripeFootersRead")->value(), 1);
+ bool eof = false;
+ size_t total = 0;
+ while (!eof) {
+ Block block = build_file_block(schema);
+ size_t rows = 0;
+ ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
+ const auto& values = assert_cast<const ColumnTimeStampNs&>(
+ assert_cast<const ColumnNullable&>(*block.get_by_position(0).column)
+ .get_nested_column());
+ for (size_t i = 0; i < rows; ++i) {
+ EXPECT_EQ(values.get_element(i).epoch_nanos(), total + i);
+ }
+ total += rows;
+ }
+ EXPECT_EQ(total, STRIPES);
+ EXPECT_LE(profile.get_counter("OrcRowReadersCreated")->value(), 7);
+ EXPECT_EQ(profile.get_counter("OrcTimestampStripeFootersRead")->value(), STRIPES);
+
+ // A late row-id lookup must not open the footer of every preceding stripe.
+ auto sparse_reader = create_reader_for_path(file, &profile);
+ ASSERT_TRUE(sparse_reader->init(&state).ok());
+ request->row_ids = std::vector<int64_t> {STRIPES - 1};
+ const auto previous_footers = profile.get_counter("OrcTimestampStripeFootersRead")->value();
+ ASSERT_TRUE(sparse_reader->open(request).ok());
+ EXPECT_EQ(profile.get_counter("OrcTimestampStripeFootersRead")->value() - previous_footers, 1);
+ Block block = build_file_block(schema);
+ size_t rows = 0;
+ eof = false;
+ ASSERT_TRUE(sparse_reader->get_block(&block, &rows, &eof).ok());
+ EXPECT_EQ(rows, 1);
+ EXPECT_EQ(schema[0].type->to_string(*block.get_by_position(0).column, 0),
+ "1970-01-01 00:00:00.000000063");
+}
+
+TEST_F(NewOrcReaderTest, JavaNanosSiblingPreservesHistoricalCivilTime) {
+ const auto file = find_repo_file("be/test/format_v2/orc/test_data/java_timestamp_nanos.orc");
+ const std::array<std::string, 3> plain {"1800-01-01 00:00:00.123457",
+ "1969-12-31 23:59:59.500000",
+ "1970-01-01 00:00:00.000000"};
+ const std::array<std::string, 3> nanos {"1800-01-01 00:00:00.123456789",
+ "1969-12-31 23:59:59.500000123",
+ "1970-01-01 00:00:00.000000001"};
+ for (const auto& zone : {"UTC", "Asia/Shanghai", "America/New_York"}) {
+ for (bool project_ns : {false, true}) {
+ auto reader = create_reader_for_path(file);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ state.set_timezone(zone);
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(0)};
+ if (project_ns) {
+ request->non_predicate_columns.push_back(field_projection(1));
+ } else {
+ schema.resize(1);
+ }
+ ASSERT_TRUE(reader->open(request).ok());
+ size_t row = 0;
+ bool eof = false;
+ while (!eof) {
+ Block block = build_file_block(schema);
+ size_t rows = 0;
+ auto status = reader->get_block(&block, &rows, &eof);
+ ASSERT_TRUE(status.ok()) << status;
+ for (size_t i = 0; i < rows; ++i, ++row) {
+ ASSERT_LT(row, plain.size());
+ EXPECT_EQ(schema[0].type->to_string(*block.get_by_position(0).column, i),
+ plain[row]);
+ if (project_ns) {
+ EXPECT_EQ(schema[1].type->to_string(*block.get_by_position(1).column, i),
+ nanos[row]);
+ }
+ }
+ }
+ EXPECT_EQ(row, plain.size());
+ }
+ }
+}
+
+TEST_F(NewOrcReaderTest, NanosecondFilesPreserveHistoricalWriterCivilTime) {
+ const auto file_path = (_test_dir / "timestamp_ns_historical.orc").string();
+ auto type = ::orc::createStructType();
+ type->addStructField("id", ::orc::createPrimitiveType(::orc::INT));
+ auto ns_type = ::orc::createPrimitiveType(::orc::TIMESTAMP);
+ ns_type->setAttribute("iceberg.timestamp-unit", "NANOS");
+ type->addStructField("ns", std::move(ns_type));
+ type->addStructField("plain", ::orc::createPrimitiveType(::orc::TIMESTAMP));
+ type->addStructField("instant", ::orc::createPrimitiveType(::orc::TIMESTAMP_INSTANT));
+ MemoryOutputStream stream(1024 * 1024);
+ ::orc::WriterOptions options;
+ options.setTimezoneName("America/New_York");
+ options.setCompression(::orc::CompressionKind_NONE);
+ // Flush each tiny batch even while all encoded values still fit in the RLE buffers.
+ options.setStripeSize(0);
+ auto writer = ::orc::createWriter(*type, &stream, options);
+ // The SDK accepts UTC seconds. These cover the historical LMT offset, the second
+ // preceding a DST fold, and different fractions at the same negative epoch second.
+ const std::array<int64_t, 4> seconds {-5364644638, -5767201, 1, 1};
+ const std::array<int64_t, 4> nanos {123456789, 500000000, 500000123, 1};
+ const std::array<std::string, 4> expected_ns {
+ "1800-01-01 00:00:00.123456789", "1969-10-26 01:59:59.500000000",
+ "1969-12-31 19:00:01.500000123", "1969-12-31 19:00:01.000000001"};
+ const std::array<std::string, 4> expected_plain {
+ "1800-01-01 00:00:00.123457", "1969-10-26 01:59:59.500000",
+ "1969-12-31 19:00:01.500000", "1969-12-31 19:00:01.000000"};
+ const std::array<std::string, 4> expected_instant {
+ "1800-01-01 12:56:02.123457", "1969-10-26 13:59:59.500000",
+ "1970-01-01 08:00:01.500000", "1970-01-01 08:00:01.000000"};
+ for (size_t start = 0; start < seconds.size(); start += 2) {
+ auto batch = writer->createRowBatch(2);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ root.numElements = 2;
+ auto& ids = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[0]);
+ ids.numElements = 2;
+ for (size_t row = 0; row < 2; ++row) {
+ ids.data[row] = static_cast<int64_t>(start + row);
+ for (size_t field = 1; field < root.fields.size(); ++field) {
+ auto& ts = dynamic_cast<::orc::TimestampVectorBatch&>(*root.fields[field]);
+ ts.numElements = 2;
+ ts.data[row] = seconds[start + row];
+ ts.nanoseconds[row] = nanos[start + row];
+ }
+ }
+ writer->add(*batch);
+ }
+ writer->close();
+ {
+ std::ofstream out(file_path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ }
+ ASSERT_EQ(get_orc_stripe_count(file_path), 2);
+ TFileScanRangeParams params;
+ params.__set_file_type(TFileType::FILE_S3);
+ params.__set_format_type(TFileFormatType::FORMAT_ORC);
+ TFileRangeDesc range;
+ range.__set_file_type(TFileType::FILE_LOCAL);
+ range.__set_path(file_path);
+ std::vector<format::ColumnDefinition> tvf_schema;
+ ASSERT_TRUE(format::fetch_file_schema(params, range, std::make_shared<io::IOContext>(), nullptr,
+ &tvf_schema)
+ .ok());
+ ASSERT_EQ(tvf_schema.size(), 4);
+ EXPECT_EQ(tvf_schema[1].type->get_primitive_type(), TYPE_TIMESTAMP_NS);
+ auto aggregate_reader = create_reader_for_path(file_path);
+ RuntimeState aggregate_state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(aggregate_reader->init(&aggregate_state).ok());
+ auto plain_request = std::make_shared<format::FileScanRequest>();
+ plain_request->non_predicate_columns = {field_projection(2)};
+ ASSERT_TRUE(aggregate_reader->open(plain_request).ok());
+ format::FileAggregateRequest aggregate_request;
+ aggregate_request.agg_type = TPushAggOp::type::MINMAX;
+ aggregate_request.columns.push_back({.projection = field_projection(2)});
+ format::FileAggregateResult aggregate_result;
+ EXPECT_TRUE(aggregate_reader->get_aggregate_result(aggregate_request, &aggregate_result)
+ .is<ErrorCode::NOT_IMPLEMENTED_ERROR>());
+ for (bool project_ns : {false, true}) {
+ for (int mode : {0, 1, 2, 3}) {
+ const bool lazy = mode == 1 || mode == 3;
+ for (int threshold : {-1, 0, 2}) {
+ RuntimeProfile profile("timestamp_stripe_ranges");
+ auto reader = create_reader_for_path(file_path, &profile);
+ TQueryOptions query_options;
+ query_options.__set_enable_orc_lazy_mat(lazy);
+ RuntimeState state {query_options, TQueryGlobals()};
+ state.set_timezone("+08:00");
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ if (mode != 3) {
+ // Isolate the lazy callback from integer stripe pruning, then test both together.
+ request->metadata_pruning_safe_conjunct_count = 0;
+ }
+ request->predicate_columns = {field_projection(0)};
+ request->non_predicate_columns =
+ project_ns ? decltype(request->non_predicate_columns) {field_projection(1),
+ field_projection(2)}
+ : decltype(request->non_predicate_columns) {field_projection(2)};
+ request->non_predicate_columns.push_back(field_projection(3));
+ request->local_positions = {
+ {format::LocalColumnId(0), format::LocalIndex(0)},
+ {format::LocalColumnId(2), format::LocalIndex(project_ns ? 2 : 1)}};
+ request->local_positions.emplace(format::LocalColumnId(3),
+ format::LocalIndex(project_ns ? 3 : 2));
+ if (project_ns) {
+ request->local_positions.emplace(format::LocalColumnId(1),
+ format::LocalIndex(1));
+ }
+ if (mode == 2) {
+ request->predicate_columns.clear();
+ request->non_predicate_columns.insert(request->non_predicate_columns.begin(),
+ field_projection(0));
+ request->row_ids = std::vector<int64_t> {};
+ for (int row = threshold + 1; row < static_cast<int>(seconds.size()); ++row) {
+ request->row_ids->push_back(row);
+ }
+ } else {
+ request->conjuncts.push_back(VExprContext::create_shared(
+ std::make_shared<NullableInt32GreaterThanExpr>(0, threshold)));
+ }
+ ASSERT_TRUE(reader->open(request).ok());
+ auto block_schema = project_ns ? schema
+ : std::vector<format::ColumnDefinition> {
+ schema[0], schema[2], schema[3]};
+ Block block = build_file_block(block_schema);
+ bool eof = false;
+ size_t output_row = static_cast<size_t>(threshold + 1);
+ while (!eof) {
+ size_t rows = 0;
+ auto status = reader->get_block(&block, &rows, &eof);
+ ASSERT_TRUE(status.ok()) << status;
+ for (size_t row = 0; row < rows; ++row, ++output_row) {
+ ASSERT_LT(output_row, seconds.size());
+ if (project_ns) {
+ EXPECT_EQ(schema[1].type->to_string(*block.get_by_position(1).column,
+ row),
+ expected_ns[output_row]);
+ }
+ EXPECT_EQ(schema[2].type->to_string(
+ *block.get_by_position(project_ns ? 2 : 1).column, row),
+ expected_plain[output_row]);
+ // An instant sibling must retain the session zone when plain values use the writer zone.
+ EXPECT_EQ(schema[3].type->to_string(
+ *block.get_by_position(project_ns ? 3 : 2).column, row),
+ expected_instant[output_row]);
+ }
+ }
+ EXPECT_EQ(output_row, seconds.size());
+ // The first stripe stays independent so LIMIT does not read later footers.
+ EXPECT_LE(profile.get_counter("OrcRowReadersCreated")->value(), 2);
+ const auto& stats = reader->reader_statistics();
+ EXPECT_EQ(stats.lazy_read_filtered_rows + (lazy ? stats.filtered_group_rows : 0),
+ lazy ? threshold + 1 : 0);
+ }
+ }
+ }
+}
+
Status decode_orc_timestamp_boundary(int64_t seconds, int64_t nanoseconds, bool file_is_instant,
bool use_timestamp_tz,
const cctz::time_zone& timezone = cctz::utc_time_zone(),
@@ -10626,11 +11018,12 @@ Status decode_orc_timestamp_boundary(int64_t seconds, int64_t nanoseconds, bool
batch.numElements = 1;
batch.data[0] = seconds;
batch.nanoseconds[0] = nanoseconds;
+ const OrcTimestampDecodeContext timestamp_context(timezone);
OrcDecodedColumnView view {.file_type = type.get(),
.selected_type = type.get(),
.batch = &batch,
.rows = 1,
- .timezone = &timezone,
+ .timestamp_context = ×tamp_context,
.enable_mapping_timestamp_tz = use_timestamp_tz};
if (use_timestamp_tz) {
DataTypeTimeStampTz data_type(6);
@@ -11922,6 +12315,102 @@ TEST_F(NewOrcReaderTest, DateCountValidatesScalarAndNestedValues) {
}
}
+TEST_F(NewOrcReaderTest, NanosCountValidatesScalarAndNestedValues) {
+ const std::vector<std::vector<std::optional<int64_t>>> cases = {
+ {-14831769600, 0, 1}, {0, 1, 2}, {std::nullopt, std::nullopt, std::nullopt}};
+ for (size_t case_id = 0; case_id < cases.size(); ++case_id) {
+ SCOPED_TRACE(case_id);
+ const auto& days = cases[case_id];
+ const auto path = (_test_dir / "count_nested_nanos.orc").string();
+ auto type = std::unique_ptr<::orc::Type>(
+ ::orc::Type::buildTypeFromString("struct<d:timestamp,a:array<timestamp>,s:struct<d:"
+ "timestamp>,m:map<int,timestamp>>"));
+ const auto annotate = [&](auto&& self, ::orc::Type& node) -> void {
+ if (node.getKind() == ::orc::TIMESTAMP) {
+ node.setAttribute("iceberg.timestamp-unit", "NANOS");
+ }
+ for (uint64_t i = 0; i < node.getSubtypeCount(); ++i) {
+ self(self, *const_cast<::orc::Type*>(node.getSubtype(i)));
+ }
+ };
+ annotate(annotate, *type);
+ MemoryOutputStream stream(1024 * 1024);
+ auto writer = ::orc::createWriter(*type, &stream, ::orc::WriterOptions());
+ auto batch = writer->createRowBatch(days.size());
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto fill = [&](auto&& self, ::orc::ColumnVectorBatch& column) -> void {
+ column.numElements = days.size();
+ if (auto* record = dynamic_cast<::orc::StructVectorBatch*>(&column)) {
+ for (auto* field : record->fields) {
+ self(self, *field);
+ }
+ } else if (auto* list = dynamic_cast<::orc::ListVectorBatch*>(&column)) {
+ for (size_t i = 0; i <= days.size(); ++i) {
+ list->offsets[i] = i;
+ }
+ self(self, *list->elements);
+ } else if (auto* map = dynamic_cast<::orc::MapVectorBatch*>(&column)) {
+ for (size_t i = 0; i <= days.size(); ++i) {
+ map->offsets[i] = i;
+ }
+ auto& keys = dynamic_cast<::orc::LongVectorBatch&>(*map->keys);
+ keys.numElements = days.size();
+ for (size_t i = 0; i < days.size(); ++i) {
+ keys.data[i] = i;
+ }
+ self(self, *map->elements);
+ } else {
+ auto& dates = dynamic_cast<::orc::TimestampVectorBatch&>(column);
+ dates.hasNulls = true;
+ for (size_t i = 0; i < days.size(); ++i) {
+ dates.notNull[i] = days[i].has_value();
+ dates.data[i] = days[i].value_or(0);
+ dates.nanoseconds[i] = 0;
+ }
+ }
+ };
+ fill(fill, root);
+ writer->add(*batch);
+ writer->close();
+ std::ofstream out(path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ out.close();
+
+ for (int column_id = 0; column_id < 4; ++column_id) {
+ SCOPED_TRACE(column_id);
+ auto reader = create_reader_for_path(path);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {field_projection(column_id)};
+ ASSERT_TRUE(reader->open(request).ok());
+ format::FileAggregateRequest aggregate;
+ aggregate.agg_type = TPushAggOp::type::COUNT;
+ format::FileAggregateResult result;
+ ASSERT_TRUE(reader->get_aggregate_result(aggregate, &result).ok());
+ EXPECT_EQ(result.count, 3);
+ aggregate.columns.push_back({.projection = field_projection(column_id)});
+ const auto status = reader->get_aggregate_result(aggregate, &result);
+ const bool invalid = case_id == 0;
+ // Millisecond ORC statistics cannot prove signed-nanos validity for every leaf.
+ EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) << status;
+ auto block = build_file_block({schema[column_id]});
+ size_t rows = 0;
+ bool eof = false;
+ const auto scan_status = reader->get_block(&block, &rows, &eof);
+ if (invalid) {
+ EXPECT_TRUE(scan_status.is<ErrorCode::DATA_QUALITY_ERROR>()) << scan_status;
+
+ } else {
+ ASSERT_TRUE(scan_status.ok()) << scan_status;
+ EXPECT_EQ(rows, 3);
+ }
+ }
+ }
+}
+
TEST_F(NewOrcReaderTest, DateSargGuardIgnoresUnprojectedStructChild) {
const auto path = (_test_dir / "partial_struct_date.orc").string();
auto type = std::unique_ptr<::orc::Type>(
@@ -11976,66 +12465,90 @@ TEST_F(NewOrcReaderTest, DateSargGuardIgnoresUnprojectedStructChild) {
}
TEST_F(NewOrcReaderTest, DateSargRetainsUnsafeStripeWithoutDisablingSafeStripePruning) {
- const auto path = (_test_dir / "mixed_date_stripes.orc").string();
- auto type = std::unique_ptr<::orc::Type>(
- ::orc::Type::buildTypeFromString("struct<id:int,d:date,payload:string>"));
- MemoryOutputStream stream(4 * 1024 * 1024);
- ::orc::WriterOptions writer_options;
- writer_options.setStripeSize(1);
- writer_options.setCompression(::orc::CompressionKind_NONE);
- writer_options.setDictionaryKeySizeThreshold(0);
- auto writer = ::orc::createWriter(*type, &stream, writer_options);
- // The safe stripes cannot match id > 7, but the invalid middle stripe must be decoded.
- for (const int64_t day : {0, -719469, 1}) {
- auto batch = writer->createRowBatch(200);
- auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
- auto& ids = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[0]);
- auto& dates = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[1]);
- auto& payload = dynamic_cast<::orc::StringVectorBatch&>(*root.fields[2]);
- std::vector<std::string> values;
- values.reserve(200);
- root.numElements = ids.numElements = dates.numElements = payload.numElements = 200;
- for (int row = 0; row < 200; ++row) {
- ids.data[row] = 0;
- dates.data[row] = day;
- values.emplace_back(2048, static_cast<char>('a' + row % 26));
- set_string_value(payload, row, values.back());
+ for (bool nanos : {false, true}) {
+ SCOPED_TRACE(nanos);
+ const auto path = (_test_dir / "mixed_date_stripes.orc").string();
+ auto type = std::unique_ptr<::orc::Type>(::orc::Type::buildTypeFromString(
+ nanos ? "struct<id:int,d:date,payload:string,ts:timestamp>"
+ : "struct<id:int,d:date,payload:string>"));
+ if (nanos) {
+ const_cast<::orc::Type*>(type->getSubtype(3))
+ ->setAttribute("iceberg.timestamp-unit", "NANOS");
+ }
+ MemoryOutputStream stream(4 * 1024 * 1024);
+ ::orc::WriterOptions writer_options;
+ writer_options.setStripeSize(1);
+ writer_options.setCompression(::orc::CompressionKind_NONE);
+ writer_options.setDictionaryKeySizeThreshold(0);
+ auto writer = ::orc::createWriter(*type, &stream, writer_options);
+ // The safe stripes cannot match id > 7, but the invalid middle stripe must be decoded.
+ for (const int64_t day : {0, -719469, 1}) {
+ auto batch = writer->createRowBatch(200);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto& ids = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[0]);
+ auto& dates = dynamic_cast<::orc::LongVectorBatch&>(*root.fields[1]);
+ auto& payload = dynamic_cast<::orc::StringVectorBatch&>(*root.fields[2]);
+ std::vector<std::string> values;
+ values.reserve(200);
+ root.numElements = ids.numElements = dates.numElements = payload.numElements = 200;
+ for (int row = 0; row < 200; ++row) {
+ ids.data[row] = 0;
+ dates.data[row] = day;
+ values.emplace_back(2048, static_cast<char>('a' + row % 26));
+ set_string_value(payload, row, values.back());
+ }
+ if (nanos) {
+ auto& timestamps = dynamic_cast<::orc::TimestampVectorBatch&>(*root.fields[3]);
+ timestamps.numElements = 200;
+ for (int row = 0; row < 200; ++row) {
+ timestamps.data[row] = 1704164645;
+ timestamps.nanoseconds[row] = 123456789;
+ }
+ }
+ writer->add(*batch);
}
- writer->add(*batch);
- }
- writer->close();
- std::ofstream out(path, std::ios::binary);
- out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
- out.close();
- ASSERT_EQ(get_orc_stripe_count(path), 3);
+ writer->close();
+ std::ofstream out(path, std::ios::binary);
+ out.write(stream.getData(), static_cast<std::streamsize>(stream.getLength()));
+ out.close();
+ ASSERT_EQ(get_orc_stripe_count(path), 3);
- for (bool pruning : {false, true}) {
- SCOPED_TRACE(pruning);
- auto reader = create_reader_for_path(path);
- TQueryOptions options;
- options.__set_enable_orc_filter_by_min_max(pruning);
- options.__set_enable_orc_lazy_mat(false);
- RuntimeState state {options, TQueryGlobals()};
- ASSERT_TRUE(reader->init(&state).ok());
- std::vector<format::ColumnDefinition> schema;
- ASSERT_TRUE(reader->get_schema(&schema).ok());
- auto request = std::make_shared<format::FileScanRequest>();
- request->predicate_columns = {field_projection(0)};
- request->non_predicate_columns = {field_projection(1)};
- request->conjuncts = {
- VExprContext::create_shared(std::make_shared<NullableInt32GreaterThanExpr>(0, 7))};
- ASSERT_TRUE(reader->open(request).ok());
- EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, pruning ? 2 : 0);
- auto block = build_file_block({schema[0], schema[1]});
- bool eof = false;
- Status status;
- while (!eof && status.ok()) {
- block.clear_column_data();
- size_t rows = 0;
- status = reader->get_block(&block, &rows, &eof);
+ for (bool pruning : {false, true}) {
+ SCOPED_TRACE(pruning);
+ auto reader = create_reader_for_path(path);
+ TQueryOptions options;
+ options.__set_enable_orc_filter_by_min_max(pruning);
+ options.__set_enable_orc_lazy_mat(false);
+ RuntimeState state {options, TQueryGlobals()};
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->predicate_columns = {field_projection(0)};
+ request->non_predicate_columns = {field_projection(1)};
+ if (nanos) {
+ // Writer-timezone regrouping must retain the invalid DATE stripe and disable its SARG.
+ request->non_predicate_columns.push_back(field_projection(3));
+ }
+ request->conjuncts = {VExprContext::create_shared(
+ std::make_shared<NullableInt32GreaterThanExpr>(0, 7))};
+ ASSERT_TRUE(reader->open(request).ok());
+ EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, pruning ? 2 : 0);
+ auto columns = std::vector<format::ColumnDefinition> {schema[0], schema[1]};
+ if (nanos) {
+ columns.push_back(schema[3]);
+ }
+ auto block = build_file_block(columns);
+ bool eof = false;
+ Status status;
+ while (!eof && status.ok()) {
+ block.clear_column_data();
+ size_t rows = 0;
+ status = reader->get_block(&block, &rows, &eof);
+ }
+ EXPECT_TRUE(status.is<ErrorCode::DATA_QUALITY_ERROR>()) << status;
+ EXPECT_NE(status.to_string().find("-719469"), std::string::npos);
}
- EXPECT_TRUE(status.is<ErrorCode::DATA_QUALITY_ERROR>()) << status;
- EXPECT_NE(status.to_string().find("-719469"), std::string::npos);
}
}
diff --git a/be/test/format_v2/orc/orc_timestamp_projection_test.cpp b/be/test/format_v2/orc/orc_timestamp_projection_test.cpp
new file mode 100644
index 00000000000..575bceb8ca1
--- /dev/null
+++ b/be/test/format_v2/orc/orc_timestamp_projection_test.cpp
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <unistd.h>
+
+#include <filesystem>
+#include <orc/OrcFile.hh>
+
+#include "core/block/block.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_struct.h"
+#include "format_v2/orc/orc_reader.h"
+#include "io/io_common.h"
+#include "runtime/runtime_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/timezone_utils.h"
+
+namespace doris {
+
+class OrcTimestampProjectionTest : public testing::Test {
+protected:
+ void SetUp() override {
+ TimezoneUtils::load_timezones_to_cache();
+ path = (std::filesystem::temp_directory_path() /
+ ("orc_timestamp_projection_" + std::to_string(getpid()) + ".orc"))
+ .string();
+ }
+ void TearDown() override { std::filesystem::remove(path); }
+ std::string path;
+};
+
+TEST_F(OrcTimestampProjectionTest, OnlyProjectedTimestampLeavesReadWriterTimezones) {
+ auto type = ::orc::Type::buildTypeFromString(
+ "struct<ns:timestamp,s:struct<i:int,unused:timestamp>>");
+ const_cast<::orc::Type*>(type->getSubtype(0))->setAttribute("iceberg.timestamp-unit", "NANOS");
+ auto output = ::orc::writeLocalFile(path);
+ ::orc::WriterOptions options;
+ options.setTimezoneName("UTC");
+ options.setStripeSize(0);
+ auto writer = ::orc::createWriter(*type, output.get(), options);
+ constexpr size_t stripes = 32;
+ for (size_t i = 0; i < stripes; ++i) {
+ auto batch = writer->createRowBatch(1);
+ auto& root = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto& ns = dynamic_cast<::orc::TimestampVectorBatch&>(*root.fields[0]);
+ auto& record = dynamic_cast<::orc::StructVectorBatch&>(*root.fields[1]);
+ auto& number = dynamic_cast<::orc::LongVectorBatch&>(*record.fields[0]);
+ auto& timestamp = dynamic_cast<::orc::TimestampVectorBatch&>(*record.fields[1]);
+ root.numElements = ns.numElements = record.numElements = number.numElements =
+ timestamp.numElements = 1;
+ number.data[0] = i;
+ ns.data[0] = timestamp.data[0] = 1704153600;
+ ns.nanoseconds[0] = timestamp.nanoseconds[0] = 123456000;
+ writer->add(*batch);
+ }
+ writer->close();
+ output->close();
+
+ for (bool project_timestamp : {false, true}) {
+ for (bool predicate_column : {false, true}) {
+ SCOPED_TRACE(testing::Message() << project_timestamp << ", " << predicate_column);
+ RuntimeProfile profile("timestamp_projection");
+ auto properties = std::make_shared<io::FileSystemProperties>();
+ properties->system_type = TFileType::FILE_LOCAL;
+ auto description = std::make_unique<io::FileDescription>();
+ description->path = path;
+ description->file_size = std::filesystem::file_size(path);
+ format::orc::OrcReader reader(properties, description, nullptr, &profile);
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ ASSERT_TRUE(reader.init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader.get_schema(&schema).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ auto projection = format::LocalColumnIndex::partial_local(1);
+ projection.children = {format::LocalColumnIndex::local(0)};
+ if (project_timestamp) {
+ projection.children.push_back(format::LocalColumnIndex::local(1));
+ }
+ (predicate_column ? request->predicate_columns : request->non_predicate_columns)
+ .push_back(projection);
+ ASSERT_TRUE(reader.open(request).ok());
+ DataTypePtr projected_type = schema[1].type;
+ if (!project_timestamp) {
+ const auto& structure =
+ assert_cast<const DataTypeStruct&>(*remove_nullable(projected_type));
+ projected_type = make_nullable(std::make_shared<DataTypeStruct>(
+ DataTypes {structure.get_elements()[0]}, Strings {"i"}));
+ }
+ bool eof = false;
+ size_t total = 0;
+ while (!eof) {
+ Block block;
+ block.insert({projected_type->create_column(), projected_type, "s"});
+ size_t rows = 0;
+ auto status = reader.get_block(&block, &rows, &eof);
+ ASSERT_TRUE(status.ok()) << status;
+ const auto& record = assert_cast<const ColumnStruct&>(
+ assert_cast<const ColumnNullable&>(*block.get_by_position(0).column)
+ .get_nested_column());
+ const auto& values = assert_cast<const ColumnInt32&>(
+ assert_cast<const ColumnNullable&>(record.get_column(0))
+ .get_nested_column());
+ for (size_t row = 0; row < rows; ++row) {
+ EXPECT_EQ(values.get_element(row), total + row);
+ if (project_timestamp) {
+ const auto& struct_type = assert_cast<const DataTypeStruct&>(
+ *remove_nullable(schema[1].type));
+ EXPECT_EQ(
+ struct_type.get_elements()[1]->to_string(
+ record.get_column(1), row, DataTypeSerDe::FormatOptions {}),
+ "2024-01-02 00:00:00.123456");
+ }
+ }
+ total += rows;
+ }
+ EXPECT_EQ(total, stripes);
+ // An unread plain timestamp must not turn a nested integer projection into
+ // per-stripe timezone discovery just because another file column is NANOS.
+ EXPECT_EQ(profile.get_counter("OrcTimestampStripeFootersRead")->value(),
+ project_timestamp ? stripes : 0);
+ if (!project_timestamp) {
+ EXPECT_EQ(profile.get_counter("OrcRowReadersCreated")->value(), 1);
+ }
+ }
+ }
+}
+
+} // namespace doris
diff --git a/be/test/format_v2/orc/test_data/GenerateJavaTimestampFixture.java b/be/test/format_v2/orc/test_data/GenerateJavaTimestampFixture.java
new file mode 100644
index 00000000000..b011bfa7115
--- /dev/null
+++ b/be/test/format_v2/orc/test_data/GenerateJavaTimestampFixture.java
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.sql.Timestamp;
+import java.util.TimeZone;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.vector.TimestampColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.orc.OrcFile;
+import org.apache.orc.TypeDescription;
+import org.apache.orc.Writer;
+
+// Generate with ORC 1.9.0, hadoop-client-{api,runtime} 3.3.5, hive-storage-api 2.8.1,
+// and protobuf-java 4.34.1 on Java 17.
+// The hidden NANOS sibling exercises the matching-writer-zone decode path.
+public class GenerateJavaTimestampFixture {
+ public static void main(String[] args) throws Exception {
+ TimeZone.setDefault(TimeZone.getTimeZone("America/New_York"));
+ TypeDescription schema = TypeDescription.fromString("struct<plain:timestamp,ns:timestamp>");
+ schema.getChildren().get(1).setAttribute("iceberg.timestamp-unit", "NANOS");
+ Configuration conf = new Configuration();
+ conf.set("fs.file.impl", "org.apache.hadoop.fs.RawLocalFileSystem");
+ String[] values = {"1800-01-01 00:00:00.123456789", "1969-12-31 23:59:59.500000123",
+ "1970-01-01 00:00:00.000000001"};
+ try (Writer writer = OrcFile.createWriter(new Path(args[0]),
+ OrcFile.writerOptions(conf).setSchema(schema).overwrite(true))) {
+ VectorizedRowBatch batch = schema.createRowBatch();
+ batch.size = values.length;
+ for (int row = 0; row < values.length; row++) {
+ for (int col = 0; col < 2; col++) {
+ ((TimestampColumnVector) batch.cols[col]).set(row, Timestamp.valueOf(values[row]));
+ }
+ }
+ writer.addRowBatch(batch);
+ }
+ }
+}
diff --git a/be/test/format_v2/orc/test_data/java_timestamp_nanos.orc b/be/test/format_v2/orc/test_data/java_timestamp_nanos.orc
new file mode 100644
index 00000000000..385ff055deb
Binary files /dev/null and b/be/test/format_v2/orc/test_data/java_timestamp_nanos.orc differ
diff --git a/be/test/format_v2/parquet/native_decoder_test.cpp b/be/test/format_v2/parquet/native_decoder_test.cpp
index 7d8ca6a7176..e2f2b5e09ad 100644
--- a/be/test/format_v2/parquet/native_decoder_test.cpp
+++ b/be/test/format_v2/parquet/native_decoder_test.cpp
@@ -37,6 +37,7 @@
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_timestamptz.h"
#include "core/data_type_serde/parquet_decode_source.h"
#include "core/data_type_serde/parquet_timestamp.h"
@@ -1334,6 +1335,36 @@ TEST(ParquetV2NativeDecoderTest, NullableSparsePlainDateTimeSelectionBatchesPhys
EXPECT_EQ(statistics.hybrid_selection_null_fallback_batches, 0);
}
+TEST(ParquetV2NativeDecoderTest, NullableNanosSelectionBatchesPhysicalPayload) {
+ const std::vector<int64_t> nanos {-1, 123456789, 987654321, 1, 999999999};
+ const std::vector<uint32_t> ids {0, 1, 2, 3, 4};
+ const std::vector<uint16_t> null_runs(9, 1);
+ const std::vector<uint8_t> filter {1, 1, 1, 0, 0, 1, 1, 1, 0};
+ const ParquetDecodeContext context {.physical_type = ParquetPhysicalType::INT64,
+ .logical_type = ParquetLogicalType::TIMESTAMP,
+ .time_unit = ::doris::ParquetTimeUnit::NANOS};
+ for (bool dictionary : {false, true}) {
+ DataTypeTimeStampNs type;
+ auto column = type.create_column();
+ NullMap nulls;
+ ColumnChunkReaderStatistics statistics;
+ auto status = dictionary ? materialize_selected_dictionary_int64(
+ nanos, ids, 9, null_runs, filter, type, &column, &nulls,
+ &statistics, context)
+ : materialize_selected_plain_int64(nanos, 9, null_runs, filter,
+ type, &column, &nulls,
+ &statistics, context);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(column->size(), 6);
+ const auto& values = assert_cast<const ColumnTimeStampNs&>(*column);
+ EXPECT_EQ(values.get_element(0).epoch_nanos(), -1);
+ EXPECT_EQ(values.get_element(2).epoch_nanos(), 123456789);
+ EXPECT_EQ(values.get_element(4).epoch_nanos(), 1);
+ EXPECT_EQ(nulls, (NullMap {0, 1, 0, 1, 0, 1}));
+ EXPECT_EQ(statistics.hybrid_selection_null_fallback_batches, 0);
+ }
+}
+
TEST(ParquetV2NativeDecoderTest, NegativeNanosFloorAcrossPlainAndDictionaryTimestamps) {
const std::vector<int64_t> nanos {-1, -999, -1000, -1001, 0, 1001};
const std::vector<uint32_t> dictionary_ids {0, 1, 2, 3, 4, 5};
diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp
index a810510126a..ae195250a1d 100644
--- a/be/test/format_v2/parquet/parquet_reader_test.cpp
+++ b/be/test/format_v2/parquet/parquet_reader_test.cpp
@@ -49,6 +49,7 @@
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
#include "core/column/column_struct.h"
+#include "core/column/column_varbinary.h"
#include "core/column/column_vector.h"
#include "core/column/variant_v2/column_variant_v2.h"
#include "core/data_type/data_type_array.h"
@@ -76,6 +77,7 @@
#include "format_v2/parquet/parquet_scan.h"
#include "format_v2/parquet/reader/column_reader.h"
#include "format_v2/schema_projection.h"
+#include "format_v2/schema_reader.h"
#include "format_v2/table_reader.h"
#include "gen_cpp/Types_types.h"
#include "io/fs/buffered_reader.h"
@@ -2287,7 +2289,7 @@ TEST_F(NewParquetReaderTest, UuidPlainDictionaryNullableAndMappingMatrix) {
std::vector<format::ColumnDefinition> schema;
ASSERT_TRUE(reader->get_schema(&schema).ok());
ASSERT_EQ(schema.size(), 1);
- auto expected_type = preserve_binary_uuid ? TYPE_STRING : TYPE_UUID;
+ auto expected_type = TYPE_UUID;
EXPECT_EQ(remove_nullable(schema[0].type)->get_primitive_type(),
expected_type);
auto request = std::make_shared<format::FileScanRequest>();
@@ -2310,20 +2312,9 @@ TEST_F(NewParquetReaderTest, UuidPlainDictionaryNullableAndMappingMatrix) {
if (expected_null) {
continue;
}
- if (preserve_binary_uuid) {
- const auto value =
- column.get_nested_column().get_data_at(row);
- EXPECT_EQ(value.size, bytes.size());
- EXPECT_EQ(value.to_string(),
- std::string(reinterpret_cast<const char*>(
- bytes.data()),
- bytes.size()));
- } else {
- EXPECT_EQ(remove_nullable(schema[0].type)
- ->to_string(column.get_nested_column(),
- row),
- "00112233-4455-6677-8899-aabbccddeeff");
- }
+ EXPECT_EQ(remove_nullable(schema[0].type)
+ ->to_string(column.get_nested_column(), row),
+ "00112233-4455-6677-8899-aabbccddeeff");
}
total += rows;
}
@@ -2354,17 +2345,17 @@ TEST_F(NewParquetReaderTest, GetSchemaReturnsFileLocalColumns) {
EXPECT_EQ(remove_nullable(schema[1].type)->get_primitive_type(), TYPE_STRING);
}
-TEST_F(NewParquetReaderTest, RawByteArrayMappingFollowsV2ScanOption) {
+TEST_F(NewParquetReaderTest, RawByteArrayMappingIgnoresLegacyOption) {
write_unannotated_binary_parquet_file(_file_path);
RuntimeState state {TQueryOptions(), TQueryGlobals()};
- auto string_reader = create_reader();
- const auto string_init_status = string_reader->init(&state);
- ASSERT_TRUE(string_init_status.ok()) << string_init_status;
- std::vector<format::ColumnDefinition> string_schema;
- ASSERT_TRUE(string_reader->get_schema(&string_schema).ok());
- ASSERT_EQ(string_schema.size(), 1);
- EXPECT_EQ(remove_nullable(string_schema[0].type)->get_primitive_type(), TYPE_STRING);
+ auto default_reader = create_reader();
+ const auto default_init_status = default_reader->init(&state);
+ ASSERT_TRUE(default_init_status.ok()) << default_init_status;
+ std::vector<format::ColumnDefinition> default_schema;
+ ASSERT_TRUE(default_reader->get_schema(&default_schema).ok());
+ ASSERT_EQ(default_schema.size(), 1);
+ EXPECT_EQ(remove_nullable(default_schema[0].type)->get_primitive_type(), TYPE_VARBINARY);
RuntimeProfile binary_profile("raw_binary_mapping_profile");
auto binary_reader =
@@ -2374,7 +2365,7 @@ TEST_F(NewParquetReaderTest, RawByteArrayMappingFollowsV2ScanOption) {
std::vector<format::ColumnDefinition> binary_schema;
ASSERT_TRUE(binary_reader->get_schema(&binary_schema).ok());
ASSERT_EQ(binary_schema.size(), 1);
- // The explicit VARBINARY mapping must remain available for scans whose table contract asks for it.
+ // The legacy option cannot reinterpret raw bytes as text.
EXPECT_EQ(remove_nullable(binary_schema[0].type)->get_primitive_type(), TYPE_VARBINARY);
auto request = std::make_shared<format::FileScanRequest>();
@@ -3800,7 +3791,7 @@ TEST_F(NewParquetReaderTest, NativeDecimalAndFixedBinaryMaterializeDirectly) {
EXPECT_EQ(decimals.get_element(1), Decimal128V3(-1));
EXPECT_EQ(decimals.get_element(3), Decimal128V3(-987654321));
- const auto& fixed_values = assert_cast<const ColumnString&>(
+ const auto& fixed_values = assert_cast<const ColumnVarbinary&>(
assert_cast<const ColumnNullable&>(*output[1]).get_nested_column());
ASSERT_EQ(fixed_values.size(), ROW_COUNT);
EXPECT_EQ(fixed_values.get_data_at(0).to_string(), "ABCD");
@@ -3942,6 +3933,65 @@ TEST_F(NewParquetReaderTest, Int96TimezoneUsesCatalogPropertyInsteadOfSessionTim
EXPECT_EQ(read_first_value(std::string {}, true), "2024-12-31 16:00:00.000000");
}
+TEST_F(NewParquetReaderTest, NanosecondsSurviveWallClockProjectionAndDictionary) {
+ const std::vector<int64_t> nanos = {
+ std::numeric_limits<int64_t>::min(), -1, 0, 1, 123456788, 123456789,
+ std::numeric_limits<int64_t>::max()};
+ auto arrow_type = arrow::timestamp(arrow::TimeUnit::NANO);
+ auto array = build_timestamp_array(arrow_type, nanos);
+ auto table = arrow::Table::Make(arrow::schema({arrow::field("ts", arrow_type, true)}), {array});
+ for (bool dictionary : {false, true}) {
+ auto file = arrow::io::FileOutputStream::Open(_file_path);
+ ASSERT_TRUE(file.ok());
+ ::parquet::WriterProperties::Builder properties;
+ properties.version(::parquet::ParquetVersion::PARQUET_2_6);
+ if (dictionary) {
+ properties.enable_dictionary();
+ } else {
+ properties.disable_dictionary();
+ }
+ ASSERT_TRUE(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), *file,
+ nanos.size(), properties.build())
+ .ok());
+ ASSERT_TRUE((*file)->Close().ok());
+ auto reader = create_reader();
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ state.set_timezone("Asia/Shanghai");
+ ASSERT_TRUE(reader->init(&state).ok());
+ std::vector<format::ColumnDefinition> schema;
+ ASSERT_TRUE(reader->get_schema(&schema).ok());
+ TFileScanRangeParams params;
+ params.__set_file_type(TFileType::FILE_S3);
+ params.__set_format_type(TFileFormatType::FORMAT_PARQUET);
+ TFileRangeDesc range;
+ range.__set_file_type(TFileType::FILE_LOCAL);
+ range.__set_path(_file_path);
+ std::vector<format::ColumnDefinition> tvf_schema;
+ ASSERT_TRUE(format::fetch_file_schema(params, range, std::make_shared<io::IOContext>(),
+ nullptr, &tvf_schema)
+ .ok());
+ ASSERT_EQ(tvf_schema.size(), 1);
+ EXPECT_EQ(tvf_schema[0].type->get_primitive_type(), TYPE_TIMESTAMP_NS);
+ ASSERT_EQ(1, schema.size());
+ ASSERT_EQ(TYPE_TIMESTAMP_NS, schema[0].type->get_primitive_type());
+ auto projection = field_projection(0);
+ projection.timestamp_is_adjusted_to_utc = false;
+ auto request = std::make_shared<format::FileScanRequest>();
+ request->non_predicate_columns = {projection};
+ ASSERT_TRUE(reader->open(request).ok());
+ Block block = build_file_block(schema);
+ size_t rows = 0;
+ bool eof = false;
+ auto status = reader->get_block(&block, &rows, &eof);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(nanos.size(), rows);
+ const auto& values = nullable_nested_column<ColumnTimeStampNs>(block, 0);
+ for (size_t row = 0; row < rows; ++row) {
+ EXPECT_EQ(nanos[row], values.get_element(row).epoch_nanos());
+ }
+ }
+}
+
TEST_F(NewParquetReaderTest, Int96UsesPerColumnTimestampSemantics) {
write_int96_timestamp_parquet_file(_file_path);
@@ -5152,8 +5202,9 @@ TEST_F(NewParquetReaderTest, AllFixedWidthDictionaryTypesDecodeThroughDictionary
EXPECT_EQ(total_rows, 6);
EXPECT_EQ(profile.get_counter("RowsFilteredByDictFilter")->value(), 0);
EXPECT_EQ(profile.get_counter("DictFilterCandidateColumns")->value(), 5);
- EXPECT_EQ(profile.get_counter("DictFilterColumns")->value(), 5);
- EXPECT_EQ(profile.get_counter("DictFilterUnsupportedColumns")->value(), 0);
+ // The fixed binary file leaf now has VARBINARY semantics and stays on residual filtering.
+ EXPECT_EQ(profile.get_counter("DictFilterColumns")->value(), 4);
+ EXPECT_EQ(profile.get_counter("DictFilterUnsupportedColumns")->value(), 1);
EXPECT_EQ(profile.get_counter("DictFilterReadFailures")->value(), 0);
}
diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp
index 990546e39e7..7b10fadbb1d 100644
--- a/be/test/format_v2/parquet/parquet_schema_test.cpp
+++ b/be/test/format_v2/parquet/parquet_schema_test.cpp
@@ -107,7 +107,33 @@ TEST(ParquetSchemaTest, LogicalAnnotationsOverrideBinaryDefault) {
iceberg.set_enable_mapping_varbinary(binary_mapping);
iceberg.set_preserve_binary_uuid(true);
ASSERT_TRUE(iceberg.parse_from_thrift(schema).ok());
- EXPECT_EQ(iceberg.get_column(2)->data_type->get_primitive_type(), TYPE_STRING);
+ // Native UUID consumers no longer request a raw string carrier for table-level decoding.
+ EXPECT_EQ(iceberg.get_column(2)->data_type->get_primitive_type(), TYPE_UUID);
+ }
+}
+
+TEST(ParquetSchemaTest, BinaryMappingDoesNotDependOnLegacyFlag) {
+ tparquet::SchemaElement root;
+ root.__set_name("schema");
+ root.__set_num_children(3);
+ tparquet::SchemaElement binary;
+ binary.__set_name("bytes");
+ binary.__set_type(tparquet::Type::BYTE_ARRAY);
+ binary.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL);
+ auto text = binary;
+ text.__set_name("text");
+ text.__set_converted_type(tparquet::ConvertedType::UTF8);
+ auto fixed = binary;
+ fixed.__set_name("fixed");
+ fixed.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY);
+ fixed.__set_type_length(4);
+ for (bool flag : {false, true}) {
+ NativeFieldDescriptor descriptor;
+ descriptor.set_enable_mapping_varbinary(flag);
+ ASSERT_TRUE(descriptor.parse_from_thrift({root, binary, text, fixed}).ok());
+ EXPECT_EQ(descriptor.get_column(0)->data_type->get_primitive_type(), TYPE_VARBINARY);
+ EXPECT_EQ(descriptor.get_column(1)->data_type->get_primitive_type(), TYPE_STRING);
+ EXPECT_EQ(descriptor.get_column(2)->data_type->get_primitive_type(), TYPE_VARBINARY);
}
}
@@ -1000,7 +1026,7 @@ TEST(ParquetSchemaTest, NativeUnknownLogicalTypeRetainsPhysicalFallback) {
ASSERT_TRUE(build_parquet_column_schema(native_schema, &fields).ok());
ASSERT_EQ(fields.size(), 1);
EXPECT_TRUE(fields[0]->type_descriptor.unsupported_reason.empty());
- EXPECT_EQ(fields[0]->type_descriptor.doris_type->get_primitive_type(), TYPE_STRING);
+ EXPECT_EQ(fields[0]->type_descriptor.doris_type->get_primitive_type(), TYPE_VARBINARY);
}
TEST(ParquetSchemaTest, NativeGroupPrimitiveLogicalTypesAreRejected) {
@@ -1638,7 +1664,7 @@ TEST(ParquetSchemaTest, NativeListPreservesLegacyMapKeyValueElement) {
EXPECT_EQ(remove_nullable(column->children[0].children[0].data_type)->get_primitive_type(),
TYPE_INT);
EXPECT_EQ(remove_nullable(column->children[0].children[1].data_type)->get_primitive_type(),
- TYPE_STRING);
+ TYPE_VARBINARY);
}
} // namespace doris::format::parquet
diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp b/be/test/format_v2/table/iceberg_reader_test.cpp
index 7e796bb8e0f..b8193879e4c 100644
--- a/be/test/format_v2/table/iceberg_reader_test.cpp
+++ b/be/test/format_v2/table/iceberg_reader_test.cpp
@@ -59,9 +59,12 @@
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
+#include "core/data_type/data_type_timestamp_ns.h"
#include "core/data_type/data_type_timestamptz.h"
+#include "core/data_type/data_type_uuid.h"
#include "core/data_type/data_type_varbinary.h"
#include "core/data_type/data_type_variant_v2.h"
+#include "core/value/uuid_value.h"
#include "exec/common/endian.h"
#include "exec/scan/access_path_parser.h"
#include "exprs/runtime_filter_expr.h"
@@ -1668,6 +1671,106 @@ TEST(IcebergV2ReaderTest, PreparesIcebergNonFiniteInitialDefaults) {
}
}
+TEST(IcebergV2ReaderTest, DatePromotionToNanosecondsChecksOverflow) {
+ const auto test_dir = std::filesystem::temp_directory_path() / "iceberg_date_promotion";
+ std::filesystem::create_directories(test_dir);
+ for (bool nested : {false, true}) {
+ for (int32_t days : {19724, -354285}) {
+ const auto file_path =
+ (test_dir / (std::to_string(days) + (nested ? "_nested.parquet" : ".parquet")))
+ .string();
+ // Read an actual retained DATE file, including NULL, through field-ID schema evolution.
+ auto field = arrow::field("ts", arrow::date32(), true,
+ arrow::key_value_metadata({"PARQUET:field_id"}, {"1"}));
+ arrow::Date32Builder builder;
+ ASSERT_TRUE(builder.Append(days).ok());
+ ASSERT_TRUE(builder.AppendNull().ok());
+ std::shared_ptr<arrow::Array> values = builder.Finish().ValueOrDie();
+ if (nested) {
+ values = arrow::StructArray::Make({values}, {field}).ValueOrDie();
+ field = arrow::field("outer", values->type(), false,
+ arrow::key_value_metadata({"PARQUET:field_id"}, {"2"}));
+ }
+ auto out = arrow::io::FileOutputStream::Open(file_path).ValueOrDie();
+ ASSERT_TRUE(::parquet::arrow::WriteTable(
+ *arrow::Table::Make(arrow::schema({field}), {values}),
+ arrow::default_memory_pool(), out, 2)
+ .ok());
+ ASSERT_TRUE(out->Close().ok());
+ for (bool filtered : {false, true}) {
+ if (nested && filtered) {
+ continue;
+ }
+ for (bool strict : {false, true}) {
+ SCOPED_TRACE(testing::Message()
+ << "nested=" << nested << " days=" << days << " strict=" << strict
+ << " filtered=" << filtered);
+ TQueryOptions query_options;
+ query_options.__set_enable_strict_cast(strict);
+ RuntimeState state {query_options, TQueryGlobals()};
+ const auto type = make_nullable(std::make_shared<DataTypeTimeStampNs>());
+ std::vector<ColumnDefinition> columns {make_table_column(1, "ts", type)};
+ if (nested) {
+ auto outer = make_table_column(
+ 2, "outer",
+ std::make_shared<DataTypeStruct>(DataTypes {type}, Strings {"ts"}));
+ outer.children = columns;
+ columns = {outer};
+ }
+ auto scan_params = make_local_parquet_scan_params();
+ RuntimeProfile profile("date_promotion");
+ io::FileReaderStats file_reader_stats;
+ io::FileCacheStatistics file_cache_stats;
+ auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
+ doris::format::iceberg::IcebergTableReader reader;
+ VExprContextSPtrs conjuncts;
+ if (filtered) {
+ // A file-local filter must not hide overflow before strict table materialization.
+ auto predicate = table_function_expr(
+ "is_not_null_pred", std::make_shared<DataTypeUInt8>(), {type});
+ predicate->add_child(VSlotRef::create_shared(0, 0, 0, type, "ts"));
+ conjuncts.push_back(prepared_conjunct(&state, predicate));
+ }
+ ASSERT_TRUE(reader.init({.projected_columns = columns,
+ .conjuncts = conjuncts,
+ .format = FileFormat::PARQUET,
+ .scan_params = &scan_params,
+ .io_ctx = io_ctx,
+ .runtime_state = &state,
+ .scanner_profile = &profile})
+ .ok());
+ ASSERT_TRUE(reader.prepare_split(build_split_options(file_path)).ok());
+ Block block = build_table_block(columns);
+ bool eos = false;
+ const auto status = reader.get_block(&block, &eos);
+ if (days < 0) {
+ EXPECT_FALSE(status.ok())
+ << "out-of-range promotion must fail even in non-strict mode";
+ EXPECT_NE(status.to_string().find("TIMESTAMP_NS overflow"),
+ std::string::npos)
+ << status;
+ } else {
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(block.rows(), filtered ? 1 : 2);
+ const IColumn* result = block.get_by_position(0).column.get();
+ if (nested) {
+ result = &assert_cast<const ColumnStruct&>(
+ expect_not_null_table_column(block, 0))
+ .get_column(0);
+ }
+ EXPECT_EQ(type->to_string(*result, 0), "2024-01-02 00:00:00.000000000");
+ if (!filtered) {
+ EXPECT_TRUE(result->is_null_at(1));
+ }
+ }
+ ASSERT_TRUE(reader.close().ok());
+ }
+ }
+ }
+ }
+ std::filesystem::remove_all(test_dir);
+}
+
TEST(IcebergV2ReaderTest, AnnotateBuildsTypedNestedInitialDefault) {
const auto int_type = make_nullable(std::make_shared<DataTypeInt32>());
const auto struct_type = make_nullable(
@@ -4221,7 +4324,7 @@ TEST(IcebergV2ReaderTest, IcebergEqualityDeleteMatchesTimestampInitialDefaultFor
std::filesystem::remove_all(test_dir);
}
-TEST(IcebergV2ReaderTest, UuidBinaryCarrierAcrossDataAndEqualityDeleteFormats) {
+TEST(IcebergV2ReaderTest, UuidNativeValuesAcrossDataAndEqualityDeleteFormats) {
const auto test_dir = std::filesystem::temp_directory_path() / "iceberg_uuid_binary_carrier";
std::filesystem::remove_all(test_dir);
std::filesystem::create_directories(test_dir);
@@ -4247,9 +4350,7 @@ TEST(IcebergV2ReaderTest, UuidBinaryCarrierAcrossDataAndEqualityDeleteFormats) {
<< "data=" << static_cast<int>(data_format)
<< " delete=" << static_cast<int>(delete_format)
<< " mapping=" << mapping << " strict=" << strict);
- DataTypePtr uuid_type =
- mapping ? DataTypePtr(std::make_shared<DataTypeVarbinary>(16))
- : DataTypePtr(std::make_shared<DataTypeString>());
+ DataTypePtr uuid_type = std::make_shared<DataTypeUUID>();
std::vector<ColumnDefinition> columns = {
make_table_column(0, "id", std::make_shared<DataTypeInt32>()),
make_table_column(1, "u", uuid_type)};
@@ -4286,8 +4387,10 @@ TEST(IcebergV2ReaderTest, UuidBinaryCarrierAcrossDataAndEqualityDeleteFormats) {
ASSERT_LT(total, values.size());
EXPECT_EQ(u.is_null_at(row), !values[total].has_value());
if (values[total].has_value()) {
- EXPECT_EQ(u.get_nested_column().get_data_at(row).to_string(),
- *values[total]);
+ EXPECT_EQ(uuid_type->to_string(u.get_nested_column(), row),
+ UUIDValue::to_string(UUIDValue::from_big_endian(
+ reinterpret_cast<const uint8_t*>(
+ values[total]->data()))));
}
}
}
@@ -4344,12 +4447,9 @@ TEST(IcebergV2ReaderTest, UuidEqualityDeleteMatchesMissingColumnInitialDefault)
scan_params.__set_current_schema_id(100);
scan_params.__set_history_schema_info({external_schema(
100, {external_schema_field("id", 0),
- external_schema_field("u", 1, {}, "ABEiM0RVZneImaq7zN3u/w==",
- external_primitive_type(
- mapping ? TPrimitiveType::VARBINARY
- : TPrimitiveType::STRING,
- 16),
- true)})});
+ external_schema_field(
+ "u", 1, {}, "00112233-4455-6677-8899-aabbccddeeff",
+ external_primitive_type(TPrimitiveType::UUID), false)})});
TQueryOptions query_options;
query_options.__set_enable_strict_cast(strict);
RuntimeState state {query_options, TQueryGlobals()};
diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml b/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml
index 79e00db2f69..a0b2e33ed00 100644
--- a/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml
+++ b/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml
@@ -37,6 +37,19 @@ under the License.
</properties>
<dependencies>
+ <!-- Match preload-extensions so Hive's bundled format classes cannot shadow the test reader. -->
+ <dependency>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-hadoop-bundle</artifactId>
+ <version>${parquet.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <!-- Production supplies Thrift through the shared loader; standalone Parquet tests need it here. -->
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>java-common</artifactId>
diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiColumnValue.java b/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiColumnValue.java
index 570363cd3a3..53b38d51f2f 100644
--- a/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiColumnValue.java
+++ b/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiColumnValue.java
@@ -31,9 +31,12 @@ import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.DateObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector;
import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.io.Text;
import java.math.BigDecimal;
import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
import java.sql.Timestamp;
import java.time.Instant;
import java.time.LocalDate;
@@ -41,6 +44,7 @@ import java.time.LocalDateTime;
import java.time.ZoneId;
import java.util.List;
import java.util.Map;
+import java.util.UUID;
public class HadoopHudiColumnValue implements ColumnValue {
private ColumnType dorisType;
@@ -105,6 +109,26 @@ public class HadoopHudiColumnValue implements ColumnValue {
return (byte[]) inspectObject();
}
+ @Override
+ public UUID getUuid() {
+ // Hudi normalizes UUID schemas to FIXED(16), but older log records can still carry Text.
+ if (fieldData instanceof Text) {
+ return UUID.fromString(fieldData.toString());
+ }
+ Object value = inspectObject();
+ if (value instanceof byte[]) {
+ byte[] bytes = (byte[]) value;
+ if (bytes.length == 16) {
+ // Avro fixed UUIDs use network byte order, independently of the JNI vector layout.
+ ByteBuffer buffer = ByteBuffer.wrap(bytes);
+ return new UUID(buffer.getLong(), buffer.getLong());
+ }
+ // A binary Hive inspector also exposes original Parquet STRING values as UTF-8 bytes.
+ return UUID.fromString(new String(bytes, StandardCharsets.UTF_8));
+ }
+ return UUID.fromString(value.toString());
+ }
+
@Override
public byte getByte() {
diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java b/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java
index 85655b56016..a49f61e9f06 100644
--- a/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java
+++ b/fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java
@@ -80,6 +80,7 @@ public class HadoopHudiJniScanner extends JniScanner {
private final String[] requiredFields;
private List<Integer> requiredColumnIds;
private ColumnType[] requiredTypes;
+ private final String[] requiredTypeNames;
// Hadoop info
private RecordReader<NullWritable, ArrayWritable> reader;
@@ -111,6 +112,7 @@ public class HadoopHudiJniScanner extends JniScanner {
this.hudiColumnNames = params.get("hudi_column_names");
this.hudiColumnTypes = params.get("hudi_column_types").split("#");
+ this.requiredTypeNames = params.get("columns_types").split("#");
// Required fields will be empty when only partition fields are selected
// This is because partition fields are not stored in the data files
if (!params.get("required_fields").equals("")) {
@@ -216,19 +218,15 @@ public class HadoopHudiJniScanner extends JniScanner {
.boxed()
.collect(Collectors.toMap(i -> splitHudiColumnNames[i], i -> i));
- Map<String, String> hudiColNameToType =
- IntStream.range(0, splitHudiColumnNames.length)
- .boxed()
- .collect(Collectors.toMap(i -> splitHudiColumnNames[i], i -> hudiColumnTypes[i]));
-
requiredTypes = new ColumnType[requiredFields.length];
for (int i = 0; i < requiredFields.length; i++) {
String requiredField = requiredFields[i];
- if (!hudiColNameToType.containsKey(requiredField)) {
+ if (!hudiColNameToIdx.containsKey(requiredField)) {
throw new IllegalArgumentException(
"Required field " + requiredField + " not found in Hudi column names: " + splitHudiColumnNames);
}
- requiredTypes[i] = ColumnType.parseType(requiredField, hudiColNameToType.get(requiredField));
+ // JNI buffers must match BE's requested types; Hive carriers lose UUID logical metadata.
+ requiredTypes[i] = ColumnType.parseType(requiredField, requiredTypeNames[i]);
}
requiredColumnIds = Arrays.stream(requiredFields)
diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HudiUuidTransportTest.java b/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HudiUuidTransportTest.java
new file mode 100644
index 00000000000..ac28a9880eb
--- /dev/null
+++ b/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HudiUuidTransportTest.java
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.hudi;
+
+import org.apache.doris.common.jni.utils.OffHeap;
+import org.apache.doris.common.jni.vec.ColumnType;
+import org.apache.doris.common.jni.vec.VectorColumn;
+
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.hadoop.hive.serde2.objectinspector.StructField;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.parquet.avro.AvroParquetWriter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+public class HudiUuidTransportTest {
+ private static final UUID UUID_VALUE = UUID.fromString("fedcba98-7654-3210-8123-456789abcdef");
+
+ private byte[] uuidBytes() {
+ return ByteBuffer.allocate(16).putLong(UUID_VALUE.getMostSignificantBits())
+ .putLong(UUID_VALUE.getLeastSignificantBits()).array();
+ }
+
+ @Test
+ public void testBinaryAndTextUuidCarriers() {
+ HadoopHudiColumnValue value = new HadoopHudiColumnValue(ZoneOffset.UTC);
+ value.setField(ColumnType.parseType("id", "uuid"),
+ PrimitiveObjectInspectorFactory.writableBinaryObjectInspector);
+ BytesWritable padded = new BytesWritable(uuidBytes());
+ padded.setCapacity(64);
+ for (Object carrier : Arrays.asList(padded, new Text(UUID_VALUE.toString()),
+ new BytesWritable(UUID_VALUE.toString().getBytes(StandardCharsets.UTF_8)))) {
+ value.setRow(carrier);
+ Assert.assertEquals(UUID_VALUE, value.getUuid());
+ }
+ value.setField(ColumnType.parseType("id", "uuid"),
+ PrimitiveObjectInspectorFactory.javaByteArrayObjectInspector);
+ value.setRow(uuidBytes());
+ Assert.assertEquals(UUID_VALUE, value.getUuid());
+ value.setField(ColumnType.parseType("id", "uuid"),
+ PrimitiveObjectInspectorFactory.javaStringObjectInspector);
+ value.setRow(UUID_VALUE.toString());
+ Assert.assertEquals(UUID_VALUE, value.getUuid());
+ value.setField(ColumnType.parseType("id", "uuid"),
+ PrimitiveObjectInspectorFactory.javaByteArrayObjectInspector);
+ value.setRow(new byte[8]);
+ Assert.assertThrows(IllegalArgumentException.class, value::getUuid);
+ }
+
+ @Test
+ public void testNestedUuidThroughHudiWritableSerdeAndJniVector() throws Exception {
+ OffHeap.setTesting();
+ for (boolean fixed : new boolean[] {false, true}) {
+ Schema uuid = LogicalTypes.uuid().addToSchema(fixed
+ ? Schema.createFixed("uuid_fixed", null, null, 16) : Schema.create(Schema.Type.STRING));
+ Schema nullable = Schema.createUnion(Schema.create(Schema.Type.NULL), uuid);
+ Schema array = Schema.createArray(nullable);
+ Schema record = Schema.createRecord("uuid_record", null, null, false);
+ record.setFields(Arrays.asList(new Schema.Field("ids", array),
+ new Schema.Field("by_name", Schema.createMap(nullable)), new Schema.Field("id", nullable)));
+ Object id = fixed ? new GenericData.Fixed(uuid, uuidBytes()) : UUID_VALUE.toString();
+ GenericData.Record row = new GenericData.Record(record);
+ row.put("ids", new GenericData.Array<>(array, Arrays.asList(id, null)));
+ Map<String, Object> byName = new HashMap<>();
+ byName.put("present", id);
+ byName.put("absent", null);
+ row.put("by_name", byName);
+ row.put("id", null);
+ ParquetHiveSerDe serde = new ParquetHiveSerDe();
+ Properties properties = new Properties();
+ properties.setProperty("columns", "value");
+ properties.setProperty("columns.types", "struct<ids:array<binary>,by_name:map<string,binary>,id:binary>");
+ serde.initialize(new Configuration(false), properties);
+ StructObjectInspector inspector = (StructObjectInspector) serde.getObjectInspector();
+ StructField field = inspector.getStructFieldRef("value");
+ HadoopHudiColumnValue value = new HadoopHudiColumnValue(ZoneOffset.UTC);
+ ColumnType type = ColumnType.parseType("value", "struct<ids:array<uuid>,by_name:map<string,uuid>,id:uuid>");
+ value.setField(type, field.getFieldObjectInspector());
+ value.setRow(HoodieRealtimeRecordReaderUtils.avroToArrayWritable(row, record));
+ VectorColumn column = VectorColumn.createWritableColumn(type, 1);
+ try {
+ column.appendValue(value);
+ Map<?, ?> restored = column.getStructColumn(0, 1)[0];
+ Assert.assertEquals(Arrays.asList(UUID_VALUE, null), restored.get("ids"));
+ Assert.assertEquals(UUID_VALUE, ((Map<?, ?>) restored.get("by_name")).get("present"));
+ Assert.assertNull(((Map<?, ?>) restored.get("by_name")).get("absent"));
+ Assert.assertNull(restored.get("id"));
+ } finally {
+ column.close();
+ }
+ }
+ }
+
+ @Test
+ public void testParquetStringAndFixedUuidWithNormalizedHiveSchema() throws Exception {
+ Path directory = Files.createTempDirectory("hudi-uuid");
+ try {
+ for (boolean fixed : new boolean[] {false, true}) {
+ Path file = directory.resolve(fixed + ".parquet");
+ Schema uuid = LogicalTypes.uuid().addToSchema(fixed
+ ? Schema.createFixed("uuid_fixed", null, null, 16) : Schema.create(Schema.Type.STRING));
+ Schema record = Schema.createRecord("uuid_record", null, null, false);
+ record.setFields(Arrays.asList(new Schema.Field("id", uuid)));
+ GenericRecord row = new GenericData.Record(record);
+ row.put("id", fixed ? new GenericData.Fixed(uuid, uuidBytes()) : UUID_VALUE.toString());
+ JobConf conf = new JobConf(false);
+ conf.set("fs.file.impl", "org.apache.hadoop.fs.RawLocalFileSystem");
+ org.apache.hadoop.fs.Path hadoopPath = new org.apache.hadoop.fs.Path(file.toUri());
+ try (ParquetWriter<GenericRecord> writer = AvroParquetWriter.<GenericRecord>builder(hadoopPath)
+ .withSchema(record).withConf(conf).build()) {
+ writer.write(row);
+ }
+ conf.set("columns", "id");
+ conf.set("columns.types", "binary");
+ conf.set("hive.io.file.read.all.columns", "true");
+ Properties properties = new Properties();
+ properties.setProperty("columns", "id");
+ properties.setProperty("columns.types", "binary");
+ ParquetHiveSerDe serde = new ParquetHiveSerDe();
+ serde.initialize(conf, properties);
+ StructObjectInspector inspector = (StructObjectInspector) serde.getObjectInspector();
+ StructField field = inspector.getStructFieldRef("id");
+ try (RecordReader<NullWritable, ArrayWritable> reader = new MapredParquetInputFormat()
+ .getRecordReader(new FileSplit(hadoopPath, 0, Files.size(file), new String[0]),
+ conf, Reporter.NULL)) {
+ ArrayWritable writable = reader.createValue();
+ Assert.assertTrue(reader.next(reader.createKey(), writable));
+ HadoopHudiColumnValue value = new HadoopHudiColumnValue(ZoneOffset.UTC);
+ value.setField(ColumnType.parseType("id", "uuid"), field.getFieldObjectInspector());
+ value.setRow(inspector.getStructFieldData(serde.deserialize(writable), field));
+ Assert.assertEquals(UUID_VALUE, value.getUuid());
+ Assert.assertFalse(reader.next(reader.createKey(), writable));
+ } finally {
+ Files.deleteIfExists(file);
+ }
+ }
+ } finally {
+ try (java.util.stream.Stream<Path> files = Files.list(directory)) {
+ for (Path file : (Iterable<Path>) files::iterator) {
+ Files.deleteIfExists(file);
+ }
+ }
+ Files.delete(directory);
+ }
+ }
+
+ @Test
+ public void testScannerUsesRequestedJniTypesInsteadOfHiveCarriers() throws Exception {
+ Map<String, String> params = new HashMap<>();
+ params.put("data_file_length", "0");
+ params.put("serde", ParquetHiveSerDe.class.getName());
+ params.put("required_fields", "ids,id");
+ params.put("columns_types", "array<uuid>#uuid");
+ params.put("hudi_column_names", "id,unused,ids");
+ params.put("hudi_column_types", "binary#string#array<binary>");
+ HadoopHudiJniScanner scanner = new HadoopHudiJniScanner(1, params);
+ Method init = HadoopHudiJniScanner.class.getDeclaredMethod("initRequiredColumnsAndTypes");
+ init.setAccessible(true);
+ init.invoke(scanner);
+ Field types = HadoopHudiJniScanner.class.getDeclaredField("requiredTypes");
+ types.setAccessible(true);
+ ColumnType[] actual = (ColumnType[]) types.get(scanner);
+ Assert.assertEquals(ColumnType.Type.UUID, actual[0].getChildTypes().get(0).getType());
+ Assert.assertEquals(ColumnType.Type.UUID, actual[1].getType());
+ Method properties = HadoopHudiJniScanner.class.getDeclaredMethod("getReaderProperties");
+ properties.setAccessible(true);
+ Assert.assertEquals("binary,string,array<binary>",
+ ((Properties) properties.invoke(scanner)).getProperty("columns.types"));
+ params.put("required_fields", "");
+ params.put("columns_types", "");
+ HadoopHudiJniScanner emptyScanner = new HadoopHudiJniScanner(1, params);
+ init.invoke(emptyScanner);
+ Assert.assertEquals(0, ((ColumnType[]) types.get(emptyScanner)).length);
+ }
+}
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/BaseJdbcExecutor.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/BaseJdbcExecutor.java
index 3e4144285ac..6ae03e8b862 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/BaseJdbcExecutor.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/BaseJdbcExecutor.java
@@ -677,6 +677,14 @@ public abstract class BaseJdbcExecutor implements JdbcExecutor {
return data.getNumRows();
}
+ protected void setUuid(int parameterIndex, java.util.UUID value) throws SQLException {
+ preparedStatement.setObject(parameterIndex, value);
+ }
+
+ protected void setUuidNull(int parameterIndex) throws SQLException {
+ preparedStatement.setNull(parameterIndex, Types.OTHER);
+ }
+
protected void setTimestampTz(int parameterIndex, java.time.LocalDateTime value) throws SQLException {
// JNI supplies UTC components; valueOf would reinterpret them in the JVM timezone.
preparedStatement.setObject(parameterIndex, Timestamp.from(value.toInstant(java.time.ZoneOffset.UTC)));
@@ -713,7 +721,7 @@ public abstract class BaseJdbcExecutor implements JdbcExecutor {
preparedStatement.setObject(parameterIndex, column.getBigInteger(rowIdx));
break;
case UUID:
- preparedStatement.setObject(parameterIndex, column.getUuid(rowIdx));
+ setUuid(parameterIndex, column.getUuid(rowIdx));
break;
case FLOAT:
preparedStatement.setFloat(parameterIndex, column.getFloat(rowIdx));
@@ -770,7 +778,7 @@ public abstract class BaseJdbcExecutor implements JdbcExecutor {
preparedStatement.setNull(parameterIndex, Types.BIGINT);
break;
case UUID:
- preparedStatement.setNull(parameterIndex, Types.OTHER);
+ setUuidNull(parameterIndex);
break;
case LARGEINT:
preparedStatement.setNull(parameterIndex, Types.JAVA_OBJECT);
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/PostgreSQLJdbcExecutor.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/PostgreSQLJdbcExecutor.java
index f46a9b0303c..30c83ae001a 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/PostgreSQLJdbcExecutor.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/PostgreSQLJdbcExecutor.java
@@ -82,6 +82,11 @@ public class PostgreSQLJdbcExecutor extends BaseJdbcExecutor {
@Override
protected Object getColumnValue(int columnIndex, ColumnType type, String[] replaceStringList) throws SQLException {
switch (type.getType()) {
+ case UUID: {
+ // JNI requires UUID objects even when the driver exposes canonical text.
+ String value = resultSet.getString(columnIndex + 1);
+ return value == null ? null : java.util.UUID.fromString(value);
+ }
case BOOLEAN:
return resultSet.getObject(columnIndex + 1, Boolean.class);
case SMALLINT:
@@ -198,6 +203,13 @@ public class PostgreSQLJdbcExecutor extends BaseJdbcExecutor {
return null;
}
switch (type.getType()) {
+ case UUID: {
+ List<java.util.UUID> result = new ArrayList<>(array.size());
+ for (Object value : array) {
+ result.add(value == null ? null : java.util.UUID.fromString(value.toString()));
+ }
+ return result;
+ }
case DATE:
case DATEV2: {
List<LocalDate> result = new ArrayList<>();
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/SQLServerJdbcExecutor.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/SQLServerJdbcExecutor.java
index 91a08a75695..71f172f9531 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/SQLServerJdbcExecutor.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/SQLServerJdbcExecutor.java
@@ -70,6 +70,11 @@ public class SQLServerJdbcExecutor extends BaseJdbcExecutor {
@Override
protected Object getColumnValue(int columnIndex, ColumnType type, String[] replaceStringList) throws SQLException {
switch (type.getType()) {
+ case UUID: {
+ // JNI requires UUID objects even when the driver exposes canonical text.
+ String value = resultSet.getString(columnIndex + 1);
+ return value == null ? null : java.util.UUID.fromString(value);
+ }
case TIMESTAMPTZ: {
// getTimestamp resolves datetimeoffset's explicit offset on old and new drivers.
// Avoid probing typed getObject: legacy drivers report unsupported conversions
@@ -103,6 +108,17 @@ public class SQLServerJdbcExecutor extends BaseJdbcExecutor {
}
}
+ @Override
+ protected void setUuid(int parameterIndex, java.util.UUID value) throws SQLException {
+ // The SQL Server driver accepts GUID text rather than a java.util.UUID object.
+ preparedStatement.setString(parameterIndex, value.toString());
+ }
+
+ @Override
+ protected void setUuidNull(int parameterIndex) throws SQLException {
+ preparedStatement.setNull(parameterIndex, java.sql.Types.VARCHAR);
+ }
+
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
// setObject(Timestamp) drops the offset and serializes JVM-local fields as datetime2.
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java
index a1304a74d88..a03f5c303b5 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java
@@ -35,6 +35,7 @@ import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
@@ -51,6 +52,17 @@ public class TrinoJdbcExecutor extends BaseJdbcExecutor {
super(thriftParams);
}
+ @Override
+ protected void setUuid(int parameterIndex, java.util.UUID value) throws SQLException {
+ // Trino's JDBC driver binds UUID through VARCHAR; the sink SQL casts it back to UUID.
+ preparedStatement.setString(parameterIndex, value.toString());
+ }
+
+ @Override
+ protected void setUuidNull(int parameterIndex) throws SQLException {
+ preparedStatement.setNull(parameterIndex, Types.VARCHAR);
+ }
+
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
if (config != null && config.getTableType() == org.apache.doris.thrift.TOdbcTableType.PRESTO) {
@@ -89,6 +101,11 @@ public class TrinoJdbcExecutor extends BaseJdbcExecutor {
@Override
protected Object getColumnValue(int columnIndex, ColumnType type, String[] replaceStringList) throws SQLException {
switch (type.getType()) {
+ case UUID: {
+ // JNI requires UUID objects even when the driver exposes canonical text.
+ String value = resultSet.getString(columnIndex + 1);
+ return value == null ? null : java.util.UUID.fromString(value);
+ }
case TIMESTAMPTZ: {
if (config != null && config.getTableType() == org.apache.doris.thrift.TOdbcTableType.PRESTO) {
// PrestoDB does not implement typed getObject for zoned timestamps.
@@ -170,6 +187,13 @@ public class TrinoJdbcExecutor extends BaseJdbcExecutor {
return Collections.emptyList();
}
switch (type.getType()) {
+ case UUID: {
+ List<java.util.UUID> result = new ArrayList<>(array.size());
+ for (Object value : array) {
+ result.add(value == null ? null : java.util.UUID.fromString(value.toString()));
+ }
+ return result;
+ }
case DATE:
case DATEV2: {
List<LocalDate> result = Lists.newArrayList();
diff --git a/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcUuidSemanticsTest.java b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcUuidSemanticsTest.java
new file mode 100644
index 00000000000..9d35c5185aa
--- /dev/null
+++ b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcUuidSemanticsTest.java
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jdbc;
+
+import org.apache.doris.common.jni.vec.ColumnType;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.sql.ResultSet;
+import java.util.Arrays;
+import java.util.UUID;
+
+class JdbcUuidSemanticsTest {
+ @Test
+ void scalarUuidsUseNativeJniValues() throws Exception {
+ UUID value = UUID.fromString("00112233-4455-6677-8899-aabbccddeeff");
+ for (BaseJdbcExecutor executor : Arrays.asList(
+ Mockito.mock(PostgreSQLJdbcExecutor.class, Mockito.CALLS_REAL_METHODS),
+ Mockito.mock(SQLServerJdbcExecutor.class, Mockito.CALLS_REAL_METHODS),
+ Mockito.mock(TrinoJdbcExecutor.class, Mockito.CALLS_REAL_METHODS))) {
+ executor.resultSet = Mockito.mock(ResultSet.class);
+ Mockito.when(executor.resultSet.getString(1)).thenReturn(value.toString(), null);
+ ColumnType type = ColumnType.parseType("id", "uuid");
+ Assertions.assertEquals(value, executor.getColumnValue(0, type, new String[0]));
+ Assertions.assertNull(executor.getColumnValue(0, type, new String[0]));
+ }
+ }
+
+ @Test
+ void nestedUuidArraysPreserveValuesAndNulls() throws Exception {
+ UUID value = UUID.fromString("ffffffff-ffff-ffff-ffff-ffffffffffff");
+ for (BaseJdbcExecutor executor : Arrays.asList(
+ Mockito.mock(PostgreSQLJdbcExecutor.class, Mockito.CALLS_REAL_METHODS),
+ Mockito.mock(TrinoJdbcExecutor.class, Mockito.CALLS_REAL_METHODS))) {
+ ColumnType type = ColumnType.parseType("ids", "array<uuid>");
+ Object[] result = executor.getOutputConverter(type, "")
+ .convert(new Object[] {Arrays.asList(value.toString(), null)});
+ Assertions.assertEquals(Arrays.asList(value, null), result[0]);
+ }
+ }
+
+ @Test
+ void driversBindUuidTextAndNull() throws Exception {
+ UUID value = UUID.fromString("00112233-4455-6677-8899-aabbccddeeff");
+ for (BaseJdbcExecutor executor : Arrays.asList(
+ Mockito.mock(SQLServerJdbcExecutor.class, Mockito.CALLS_REAL_METHODS),
+ Mockito.mock(TrinoJdbcExecutor.class, Mockito.CALLS_REAL_METHODS))) {
+ executor.preparedStatement = Mockito.mock(java.sql.PreparedStatement.class);
+ executor.setUuid(1, value);
+ executor.setUuidNull(2);
+ Mockito.verify(executor.preparedStatement).setString(1, value.toString());
+ Mockito.verify(executor.preparedStatement).setNull(2, java.sql.Types.VARCHAR);
+ }
+ }
+
+}
diff --git a/fe/be-java-extensions/paimon-connector/pom.xml b/fe/be-java-extensions/paimon-connector/pom.xml
index afbc5a77dec..717b3e6f2ce 100644
--- a/fe/be-java-extensions/paimon-connector/pom.xml
+++ b/fe/be-java-extensions/paimon-connector/pom.xml
@@ -86,6 +86,32 @@ under the License.
<artifactId>arrow-c-data</artifactId>
</dependency>
+ <!-- Local Parquet/ORC round trips need Hadoop classes supplied by BE at runtime. -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs-client</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
</dependencies>
<build>
diff --git a/fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java b/fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java
index 918f4789073..d6dde7fad8a 100644
--- a/fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java
+++ b/fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonArrowBatchAdapter.java
@@ -133,16 +133,17 @@ final class PaimonArrowBatchAdapter {
@Override
public Arrow2PaimonVectorConverter visit(TimestampType timestampType) {
- return timestampAdapter();
+ return timestampAdapter(timestampType.getPrecision());
}
@Override
public Arrow2PaimonVectorConverter visit(
LocalZonedTimestampType localZonedTimestampType) {
- return timestampAdapter();
+ return timestampAdapter(localZonedTimestampType.getPrecision());
}
- private static Arrow2PaimonVectorConverter timestampAdapter() {
+ private static Arrow2PaimonVectorConverter timestampAdapter(int targetPrecision) {
+ final int nanosPerUnit = (int) Math.pow(10, 9 - targetPrecision);
return vector -> {
TimeStampVector timestamps = (TimeStampVector) vector;
TimeUnit unit = ((ArrowType.Timestamp) vector.getField().getType()).getUnit();
@@ -163,9 +164,15 @@ final class PaimonArrowBatchAdapter {
case MICROSECOND:
return Timestamp.fromMicros(value);
case NANOSECOND:
- return Timestamp.fromEpochMillis(
- Math.floorDiv(value, 1_000_000L),
- (int) Math.floorMod(value, 1_000_000L));
+ // Paimon serializers trust target precision. Truncate the positive
+ // fractional remainder so negative epochs and leading zero digits
+ // obey the same narrowing rule, including nested values and keys.
+ int nanos = (int) Math.floorMod(value, 1_000_000L);
+ // Flooring precision 7/8 can move Long.MIN_VALUE below the signed
+ // nanos range. Reject it before persisting a value JNI cannot read.
+ Math.subtractExact(value, nanos % nanosPerUnit);
+ return Timestamp.fromEpochMillis(Math.floorDiv(value, 1_000_000L),
+ nanos / nanosPerUnit * nanosPerUnit);
default:
throw new IllegalArgumentException(
"Unsupported Paimon Arrow timestamp unit: " + unit);
@@ -204,6 +211,10 @@ final class PaimonArrowBatchAdapter {
TimeUnit unit = precision > 3
? TimeUnit.MICROSECOND
: precision > 0 ? TimeUnit.MILLISECOND : TimeUnit.SECOND;
+ if (timeZone == null && precision > 6) {
+ // The transport must retain the sub-microsecond digits of TIMESTAMP_NS inputs.
+ unit = TimeUnit.NANOSECOND;
+ }
return new FieldType(nullable, new ArrowType.Timestamp(unit, timeZone), null);
}
}
diff --git a/fe/be-java-extensions/paimon-connector/src/test/java/org/apache/doris/paimon/PaimonArrowBatchAdapterTest.java b/fe/be-java-extensions/paimon-connector/src/test/java/org/apache/doris/paimon/PaimonArrowBatchAdapterTest.java
index 73c8685b20e..ba07f0e2ed5 100644
--- a/fe/be-java-extensions/paimon-connector/src/test/java/org/apache/doris/paimon/PaimonArrowBatchAdapterTest.java
+++ b/fe/be-java-extensions/paimon-connector/src/test/java/org/apache/doris/paimon/PaimonArrowBatchAdapterTest.java
@@ -40,14 +40,25 @@ import org.apache.paimon.data.InternalMap;
import org.apache.paimon.data.InternalRow;
import org.apache.paimon.data.variant.GenericVariant;
import org.apache.paimon.data.variant.Variant;
+import org.apache.paimon.fs.local.LocalFileIO;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.table.FileStoreTableFactory;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.table.sink.BatchTableCommit;
+import org.apache.paimon.table.sink.BatchTableWrite;
+import org.apache.paimon.table.sink.BatchWriteBuilder;
+import org.apache.paimon.table.source.ReadBuilder;
import org.apache.paimon.types.DataType;
import org.apache.paimon.types.DataTypes;
import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.CloseableIterator;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
import java.io.ByteArrayInputStream;
import java.math.BigDecimal;
+import java.nio.file.Path;
import java.time.ZoneId;
import java.util.Arrays;
import java.util.LinkedHashMap;
@@ -174,9 +185,7 @@ public class PaimonArrowBatchAdapterTest {
Assertions.assertNull(ntz.getTimezone());
Assertions.assertEquals(TimeUnit.MICROSECOND, ltz.getUnit());
Assertions.assertEquals("Asia/Shanghai", ltz.getTimezone());
- // Doris timestamps have microsecond precision, so a wider Paimon target still
- // uses a microsecond transport instead of pretending the input contains nanos.
- Assertions.assertEquals(TimeUnit.MICROSECOND, ntz9.getUnit());
+ Assertions.assertEquals(TimeUnit.NANOSECOND, ntz9.getUnit());
Assertions.assertNull(ntz9.getTimezone());
}
}
@@ -199,13 +208,144 @@ public class PaimonArrowBatchAdapterTest {
root.allocateNew();
((TimeStampVector) root.getVector("ntz")).setSafe(0, ntzMicros);
((TimeStampVector) root.getVector("ltz")).setSafe(0, ltzInstantMicros);
- ((TimeStampVector) root.getVector("ntz9")).setSafe(0, ntzMicros);
+ ((TimeStampVector) root.getVector("ntz9")).setSafe(0, ntzMicros * 1_000L + 789);
root.setRowCount(1);
InternalRow row = adapter.rows(root).row(0);
Assertions.assertEquals(ntzMicros, row.getTimestamp(0, 6).toMicros());
Assertions.assertEquals(ltzInstantMicros, row.getTimestamp(1, 6).toMicros());
Assertions.assertEquals(ntzMicros, row.getTimestamp(2, 9).toMicros());
+ Assertions.assertEquals(456789, row.getTimestamp(2, 9).getNanoOfMillisecond());
+ }
+ }
+ }
+
+ @Test
+ public void testNanosecondTransportRespectsTargetPrecision() throws Exception {
+ for (int precision : new int[] {7, 8, 9}) {
+ RowType inputType = DataTypes.ROW(DataTypes.FIELD(0, "ts", DataTypes.TIMESTAMP(precision)),
+ DataTypes.FIELD(1, "nested", DataTypes.ROW(
+ DataTypes.FIELD(0, "ts", DataTypes.TIMESTAMP(precision)))));
+ try (RootAllocator allocator = new RootAllocator()) {
+ PaimonArrowBatchAdapter adapter = new PaimonArrowBatchAdapter(inputType, ZoneId.of("UTC"), allocator);
+ try (ArrowStreamReader reader = schemaReader(adapter, allocator)) {
+ VectorSchemaRoot root = reader.getVectorSchemaRoot();
+ root.allocateNew();
+ StructVector nested = (StructVector) root.getVector("nested");
+ for (long nanos : new long[] {123456789, -123456789, -1, 0, 1}) {
+ ((TimeStampVector) root.getVector("ts")).setSafe(0, nanos);
+ ((TimeStampVector) nested.getChild("ts")).setSafe(0, nanos);
+ nested.setIndexDefined(0);
+ root.setRowCount(1);
+ InternalRow row = adapter.rows(root).row(0);
+ long unit = precision == 7 ? 100 : precision == 8 ? 10 : 1;
+ long expected = Math.floorDiv(nanos, unit) * unit;
+ for (org.apache.paimon.data.Timestamp value : new org.apache.paimon.data.Timestamp[] {
+ row.getTimestamp(0, precision), row.getRow(1, 1).getTimestamp(0, precision)}) {
+ Assertions.assertEquals(Math.floorDiv(expected, 1_000_000L), value.getMillisecond());
+ Assertions.assertEquals(Math.floorMod(expected, 1_000_000L), value.getNanoOfMillisecond());
+ }
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testPrecisionNarrowingCannotLeaveSignedNanosRange() throws Exception {
+ for (int precision : new int[] {7, 8}) {
+ RowType inputType = DataTypes.ROW(DataTypes.FIELD(0, "ts", DataTypes.TIMESTAMP(precision)),
+ DataTypes.FIELD(1, "nested", DataTypes.ROW(
+ DataTypes.FIELD(0, "ts", DataTypes.TIMESTAMP(precision)))));
+ try (RootAllocator allocator = new RootAllocator()) {
+ PaimonArrowBatchAdapter adapter = new PaimonArrowBatchAdapter(inputType, ZoneId.of("UTC"), allocator);
+ try (ArrowStreamReader reader = schemaReader(adapter, allocator)) {
+ VectorSchemaRoot root = reader.getVectorSchemaRoot();
+ root.allocateNew();
+ StructVector nested = (StructVector) root.getVector("nested");
+ ((TimeStampVector) root.getVector("ts")).setSafe(0, Long.MIN_VALUE);
+ ((TimeStampVector) nested.getChild("ts")).setSafe(0, Long.MIN_VALUE);
+ nested.setIndexDefined(0);
+ root.setRowCount(1);
+ InternalRow row = adapter.rows(root).row(0);
+ // The input fits, but flooring its fractional precision must not create an unreadable value.
+ Assertions.assertThrows(ArithmeticException.class, () -> row.getTimestamp(0, precision));
+ Assertions.assertThrows(ArithmeticException.class,
+ () -> row.getRow(1, 1).getTimestamp(0, precision));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testPrecisionSevenAndEightKeysRoundTrip(@TempDir Path dir)
+ throws Exception {
+ for (String format : new String[] {"parquet", "orc"}) {
+ for (int precision : new int[] {7, 8}) {
+ LocalFileIO fileIO = LocalFileIO.create();
+ org.apache.paimon.fs.Path path = new org.apache.paimon.fs.Path(
+ dir.resolve(format + precision).toUri());
+ org.apache.paimon.schema.Schema schema = org.apache.paimon.schema.Schema.newBuilder()
+ .column("ts", DataTypes.TIMESTAMP(precision).notNull())
+ .column("detail", DataTypes.ROW(DataTypes.FIELD(0, "t", DataTypes.TIMESTAMP(precision))))
+ .primaryKey("ts").option("bucket", "1").option("file.format", format).build();
+ new SchemaManager(fileIO, path).createTable(schema);
+ Table table = FileStoreTableFactory.create(fileIO, path);
+ PaimonWriteSchema writeSchema = PaimonWriteSchema.create(table.rowType(), new String[] {"ts", "detail"});
+ BatchWriteBuilder builder = table.newBatchWriteBuilder();
+ try (RootAllocator allocator = new RootAllocator();
+ BatchTableWrite write = builder.newWrite();
+ BatchTableCommit commit = builder.newCommit()) {
+ PaimonArrowBatchAdapter adapter = new PaimonArrowBatchAdapter(
+ table.rowType(), ZoneId.of("UTC"), allocator);
+ try (ArrowStreamReader reader = schemaReader(adapter, allocator)) {
+ VectorSchemaRoot root = reader.getVectorSchemaRoot();
+ root.allocateNew();
+ for (long nanos : new long[] {123456781, 123456789}) {
+ ((TimeStampVector) root.getVector("ts")).setSafe(0, nanos);
+ StructVector nested = (StructVector) root.getVector("detail");
+ ((TimeStampVector) nested.getChild("t")).setSafe(0, nanos);
+ nested.setIndexDefined(0);
+ root.setRowCount(1);
+ write.write(writeSchema.tableRow(adapter.rows(root).row(0)));
+ }
+ }
+ commit.commit(write.prepareCommit());
+ }
+ ReadBuilder read = table.newReadBuilder();
+ int count = 0;
+ try (CloseableIterator<InternalRow> rows =
+ read.newRead().createReader(read.newScan().plan()).toCloseableIterator()) {
+ while (rows.hasNext()) {
+ InternalRow row = rows.next();
+ int expected = precision == 7 ? 456700 : 456780;
+ Assertions.assertEquals(expected, row.getTimestamp(0, precision).getNanoOfMillisecond());
+ Assertions.assertEquals(expected,
+ row.getRow(1, 1).getTimestamp(0, precision).getNanoOfMillisecond());
+ count++;
+ }
+ }
+ // Precision narrowing must happen before hashing and serialization of primary keys.
+ Assertions.assertEquals(1, count);
+ }
+ }
+ }
+
+ @Test
+ public void testNanosecondTransportPreservesSignedBounds() throws Exception {
+ RowType inputType = DataTypes.ROW(DataTypes.FIELD(0, "ts", DataTypes.TIMESTAMP(9)));
+ try (RootAllocator allocator = new RootAllocator()) {
+ PaimonArrowBatchAdapter adapter = new PaimonArrowBatchAdapter(inputType, ZoneId.of("UTC"), allocator);
+ try (ArrowStreamReader reader = schemaReader(adapter, allocator)) {
+ VectorSchemaRoot root = reader.getVectorSchemaRoot();
+ root.allocateNew();
+ for (long nanos : new long[] {Long.MIN_VALUE, -1, 0, 1, Long.MAX_VALUE}) {
+ ((TimeStampVector) root.getVector("ts")).setSafe(0, nanos);
+ root.setRowCount(1);
+ org.apache.paimon.data.Timestamp value = adapter.rows(root).row(0).getTimestamp(0, 9);
+ Assertions.assertEquals(Math.floorDiv(nanos, 1_000_000L), value.getMillisecond());
+ Assertions.assertEquals(Math.floorMod(nanos, 1_000_000L), value.getNanoOfMillisecond());
+ }
}
}
}
diff --git a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorColumnValue.java b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorColumnValue.java
index 69722f42bca..c9388b7b82f 100644
--- a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorColumnValue.java
+++ b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorColumnValue.java
@@ -131,6 +131,13 @@ public class TrinoConnectorColumnValue implements ColumnValue {
return Decimals.readBigDecimal((DecimalType) trinoType, block, position);
}
+ @Override
+ public java.util.UUID getUuid() {
+ // Trino stores UUIDs in a fixed-width block with its own byte-order convention.
+ return io.trino.spi.type.UuidType.trinoUuidToJavaUuid(
+ io.trino.spi.type.UuidType.UUID.getSlice(block, position));
+ }
+
// block is VariableWidthBlock
@Override
public String getString() {
diff --git a/fe/be-java-extensions/trino-connector-scanner/src/test/java/org/apache/doris/trinoconnector/TrinoTimestampSemanticsTest.java b/fe/be-java-extensions/trino-connector-scanner/src/test/java/org/apache/doris/trinoconnector/TrinoTimestampSemanticsTest.java
index 78353fa11a0..9fcb4166ce9 100644
--- a/fe/be-java-extensions/trino-connector-scanner/src/test/java/org/apache/doris/trinoconnector/TrinoTimestampSemanticsTest.java
+++ b/fe/be-java-extensions/trino-connector-scanner/src/test/java/org/apache/doris/trinoconnector/TrinoTimestampSemanticsTest.java
@@ -58,4 +58,18 @@ class TrinoTimestampSemanticsTest {
}
}
}
+
+ @Test
+ void nativeUuidUsesTrinoByteOrderConversion() {
+ java.util.UUID expected = java.util.UUID.fromString("00112233-4455-6677-8899-aabbccddeeff");
+ io.trino.spi.type.UuidType type = io.trino.spi.type.UuidType.UUID;
+ BlockBuilder block = type.createBlockBuilder(null, 1);
+ type.writeSlice(block, io.trino.spi.type.UuidType.javaUuidToTrinoUuid(expected));
+ TrinoConnectorColumnValue value = new TrinoConnectorColumnValue();
+ value.setTrinoType(type);
+ value.setBlock(block.build());
+ value.setPosition(0);
+ Assertions.assertEquals(expected, value.getUuid());
+ }
+
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/UuidLiteral.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/UuidLiteral.java
index 61ad8195d78..9be97044431 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/UuidLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/UuidLiteral.java
@@ -72,6 +72,14 @@ public class UuidLiteral extends LiteralExpr {
@Override
protected String toSqlImpl(boolean disableTableName, boolean needExternalSql, TableType tableType,
TableIf table) {
+ if (needExternalSql && table instanceof org.apache.doris.catalog.JdbcTable
+ && (((org.apache.doris.catalog.JdbcTable) table).getJdbcTableType()
+ == org.apache.doris.thrift.TOdbcTableType.TRINO
+ || ((org.apache.doris.catalog.JdbcTable) table).getJdbcTableType()
+ == org.apache.doris.thrift.TOdbcTableType.PRESTO)) {
+ // Both dialects expose native UUIDs and reject implicit VARCHAR coercion in predicates.
+ return "UUID '" + getStringValue() + "'";
+ }
return needExternalSql ? "'" + getStringValue() + "'" : toSqlImpl();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java
index bebb13e5b3b..02464f1c6d7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java
@@ -161,6 +161,10 @@ public class JdbcTable extends Table {
} else if (column != null && column.getType().isTimeStampTz() && remoteType == TOdbcTableType.PRESTO) {
sb.append("io.prestosql.jdbc.PrestoDriver".equals(getDriverClass())
? "CAST(? AS TIMESTAMP(6) WITH TIME ZONE)" : "CAST(? AS TIMESTAMP WITH TIME ZONE)");
+ } else if (column != null && column.getType().isUuid()
+ && (remoteType == TOdbcTableType.TRINO || remoteType == TOdbcTableType.PRESTO)) {
+ // UUID parameters may be bound as text; preserve the target type in remote SQL.
+ sb.append("CAST(? AS UUID)");
} else {
sb.append("?");
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiUtils.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiUtils.java
index 421caacbb61..3b78531186a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiUtils.java
@@ -100,6 +100,10 @@ public class HudiUtils {
return "string";
case FIXED:
case BYTES:
+ // Keep normalized UUID bytes intact until the JNI scanner decodes their logical type.
+ if (logicalType != null && "uuid".equals(logicalType.getName())) {
+ return "binary";
+ }
if (logicalType instanceof LogicalTypes.Decimal) {
LogicalTypes.Decimal decimalType = (LogicalTypes.Decimal) logicalType;
return String.format("decimal(%d,%d)", decimalType.getPrecision(), decimalType.getScale());
@@ -183,7 +187,8 @@ public class HudiUtils {
case DOUBLE:
return Type.DOUBLE;
case STRING:
- return Type.STRING;
+ // Avro UUID is logical metadata over STRING, not an ordinary text column.
+ return logicalType != null && "uuid".equals(logicalType.getName()) ? Type.UUID : Type.STRING;
case FIXED:
case BYTES:
return handleFixedOrBytesType(logicalType);
@@ -232,6 +237,10 @@ public class HudiUtils {
}
private static Type handleFixedOrBytesType(LogicalType logicalType) {
+ // Hudi's InternalSchema round trip rewrites Avro STRING UUIDs as annotated FIXED(16).
+ if (logicalType != null && "uuid".equals(logicalType.getName())) {
+ return Type.UUID;
+ }
if (logicalType instanceof LogicalTypes.Decimal) {
int precision = ((LogicalTypes.Decimal) logicalType).getPrecision();
int scale = ((LogicalTypes.Decimal) logicalType).getScale();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
index eba180512b4..536a3855ee4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
@@ -119,6 +119,8 @@ public class DorisTypeToIcebergType extends DorisTypeVisitor<Type> {
return Types.FloatType.get();
} else if (primitiveType.equals(PrimitiveType.DOUBLE)) {
return Types.DoubleType.get();
+ } else if (primitiveType.equals(PrimitiveType.UUID)) {
+ return Types.UUIDType.get();
} else if (primitiveType.isCharFamily()) {
return Types.StringType.get();
} else if (primitiveType.equals(PrimitiveType.DATE)
@@ -132,6 +134,8 @@ public class DorisTypeToIcebergType extends DorisTypeVisitor<Type> {
} else if (primitiveType.equals(PrimitiveType.DATETIME)
|| primitiveType.equals(PrimitiveType.DATETIMEV2)) {
return Types.TimestampType.withoutZone();
+ } else if (primitiveType.equals(PrimitiveType.TIMESTAMP_NS)) {
+ return Types.TimestampNanoType.withoutZone();
} else if (primitiveType.equals(PrimitiveType.TIMESTAMPTZ)) {
return Types.TimestampType.withZone();
} else if (primitiveType.equals(PrimitiveType.VARIANT)) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergConflictDetectionFilterUtils.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergConflictDetectionFilterUtils.java
index 5423e21895d..b1626e7c0eb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergConflictDetectionFilterUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergConflictDetectionFilterUtils.java
@@ -68,8 +68,14 @@ public final class IcebergConflictDetectionFilterUtils {
return Optional.empty();
}
Schema schema = targetTable.getIcebergTable().schema();
+ Set<Integer> unsafeNanosFields = IcebergTimestampUtils.unsafeNanosFields(
+ targetTable.getIcebergTable(), schema);
org.apache.iceberg.expressions.Expression combined = null;
for (Expression predicate : targetConjuncts) {
+ // A widened conflict filter is safe; retained DATE bounds must not hide conflicts.
+ if (IcebergTimestampUtils.referencesUnsafe(predicate, schema, unsafeNanosFields)) {
+ continue;
+ }
Optional<org.apache.iceberg.expressions.Expression> icebergExpr =
convertPredicateToIcebergExpression(predicate, schema);
if (!icebergExpr.isPresent()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergNereidsUtils.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergNereidsUtils.java
index 2308ee0b86d..80744b2e62e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergNereidsUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergNereidsUtils.java
@@ -524,8 +524,15 @@ public class IcebergNereidsUtils {
}
// accept string value for date
return literal.getStringValue();
- case TIMESTAMP:
- case TIMESTAMP_NANO: {
+ case TIMESTAMP_NANO:
+ if (((org.apache.iceberg.types.Types.TimestampNanoType) icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ // Rewrite WHERE strings have not undergone SQL coercion: normalize explicit
+ // offsets into session civil time before encoding the unzoned nanos value.
+ return Expressions.nanos(IcebergTimestampUtils.parseNanos(
+ literal.getStringValue(), DateUtils.getTimeZone()));
+ case TIMESTAMP: {
// Iceberg expects microseconds since epoch. Honor with/without zone semantics.
if (literal instanceof DateTimeLiteral
|| literal instanceof DateLiteral) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTimestampUtils.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTimestampUtils.java
new file mode 100644
index 00000000000..ca925fecdba
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTimestampUtils.java
@@ -0,0 +1,94 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.nereids.analyzer.UnboundSlot;
+import org.apache.doris.nereids.trees.expressions.literal.TimeStampNsLiteral;
+
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+import java.math.BigDecimal;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Nanosecond conversion and retained DATE-metric safeguards shared by the 4.2 planners. */
+public final class IcebergTimestampUtils {
+ private IcebergTimestampUtils() {
+ }
+
+ public static long parseNanos(String value, ZoneId zone) {
+ // Static partitions must accept and round exactly the same literals as the sink cast.
+ LocalDateTime timestamp = TimeStampNsLiteral.parseDateTime(value, zone);
+ // Widen before combining: the minimum timestamp's whole second overflows long nanos.
+ return BigDecimal.valueOf(timestamp.toEpochSecond(ZoneOffset.UTC))
+ .multiply(BigDecimal.valueOf(1_000_000_000L))
+ .add(BigDecimal.valueOf(timestamp.getNano())).longValueExact();
+ }
+
+ public static Set<Integer> unsafeNanosFields(Table table, Schema schema) {
+ Set<Integer> unsafe = new HashSet<>();
+ Map<Integer, Schema> history = table.schemas();
+ int maxId = history.keySet().stream().mapToInt(Integer::intValue).max().orElse(-1);
+ // Removed history can leave live DATE files whose four-byte bounds Iceberg reads as longs.
+ boolean incomplete = history.isEmpty() || history.size() != (long) maxId + 1;
+ for (Types.NestedField field : TypeUtil.indexById(schema.asStruct()).values()) {
+ if (field.type().typeId() != Type.TypeID.TIMESTAMP_NANO) {
+ continue;
+ }
+ if (incomplete || history.values().stream().anyMatch(old -> {
+ Type type = old.findType(field.fieldId());
+ return type != null && type.typeId() == Type.TypeID.DATE;
+ })) {
+ unsafe.add(field.fieldId());
+ }
+ }
+ return unsafe;
+ }
+
+ private static boolean unsafeName(String name, Schema schema, Set<Integer> unsafe) {
+ Types.NestedField field = schema.caseInsensitiveFindField(name);
+ return field != null && unsafe.contains(field.fieldId());
+ }
+
+ public static boolean referencesUnsafe(Expr expr, Schema schema, Set<Integer> unsafe) {
+ if (expr instanceof SlotRef && unsafeName(((SlotRef) expr).getColumnName(), schema, unsafe)) {
+ return true;
+ }
+ return expr.getChildren().stream().anyMatch(child -> referencesUnsafe(child, schema, unsafe));
+ }
+
+ public static boolean referencesUnsafe(org.apache.doris.nereids.trees.expressions.Expression expr,
+ Schema schema, Set<Integer> unsafe) {
+ return expr.getInputSlots().stream().anyMatch(slot -> {
+ // Match IcebergNereidsUtils: SQL-rendered names add quoting to literal dots/backticks.
+ String name = slot instanceof UnboundSlot && ((UnboundSlot) slot).getNameParts().size() == 1
+ ? ((UnboundSlot) slot).getNameParts().get(0) : slot.getName();
+ return unsafeName(name, schema, unsafe);
+ });
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java
index 444798b936a..3c2b40ae609 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java
@@ -78,6 +78,7 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import java.util.function.BooleanSupplier;
public class IcebergTransaction implements Transaction {
@@ -974,6 +975,7 @@ public class IcebergTransaction implements Transaction {
}
Schema schema = icebergTable.schema();
+ Set<Integer> unsafeNanosFields = IcebergTimestampUtils.unsafeNanosFields(icebergTable, schema);
int currentSpecId = spec.specId();
Expression combined = null;
@@ -991,7 +993,8 @@ public class IcebergTransaction implements Transaction {
return Optional.empty();
}
- Expression partitionExpr = buildIdentityPartitionExpression(spec, schema, partitionValues);
+ Expression partitionExpr = buildIdentityPartitionExpression(
+ spec, schema, partitionValues, unsafeNanosFields);
if (partitionExpr == null) {
return Optional.empty();
}
@@ -1010,7 +1013,7 @@ public class IcebergTransaction implements Transaction {
}
private Expression buildIdentityPartitionExpression(PartitionSpec spec, Schema schema,
- List<String> partitionValues) {
+ List<String> partitionValues, Set<Integer> unsafeNanosFields) {
Expression expression = null;
List<PartitionField> fields = spec.fields();
for (int i = 0; i < fields.size(); i++) {
@@ -1028,6 +1031,13 @@ public class IcebergTransaction implements Transaction {
valueStr = null;
}
Object value = IcebergUtils.parsePartitionValueFromString(valueStr, sourceField.type());
+ // Conflict detection may widen but must not miss retained DATE files.
+ if (value != null && unsafeNanosFields.contains(sourceField.fieldId())) {
+ continue;
+ }
+ if (value != null && sourceField.type().typeId() == org.apache.iceberg.types.Type.TypeID.TIMESTAMP_NANO) {
+ value = Expressions.nanos((Long) value);
+ }
Expression predicate = value == null
? Expressions.isNull(sourceColumnName)
: Expressions.equal(sourceColumnName, value);
@@ -1246,6 +1256,16 @@ public class IcebergTransaction implements Transaction {
Object partitionValue = IcebergUtils.parsePartitionValueFromString(
partitionValueStr, sourceField.type());
+ if (partitionValue != null
+ && sourceField.type().typeId() == org.apache.iceberg.types.Type.TypeID.TIMESTAMP_NANO) {
+ // Dropping a deletion predicate would widen a static overwrite to unrelated data.
+ if (IcebergTimestampUtils.unsafeNanosFields(table, schema).contains(sourceField.fieldId())) {
+ throw new IllegalArgumentException(
+ "Cannot overwrite timestamp_ns partition with historical DATE bounds");
+ }
+ partitionValue = Expressions.nanos((Long) partitionValue);
+ }
+
// Build equality expression using the source column path (not partition field name).
// For identity partitions, Iceberg requires the source column in expressions.
Expression eqExpr;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
index 691de0fe9ed..7d82c8dadc1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
@@ -155,7 +155,6 @@ import java.time.format.DateTimeParseException;
import java.time.temporal.ChronoField;
import java.time.temporal.TemporalAccessor;
import java.util.ArrayList;
-import java.util.Base64;
import java.util.Comparator;
import java.util.EnumMap;
import java.util.HashMap;
@@ -293,6 +292,11 @@ public class IcebergUtils {
return false;
}
PartitionField partitionField = fields.get(0);
+ org.apache.iceberg.types.Type sourceType = spec.schema().findType(partitionField.sourceId());
+ // Calendar boundaries and NULL sentinels can exceed the signed nanosecond domain.
+ if (sourceType == null || sourceType.typeId() == TypeID.TIMESTAMP_NANO) {
+ return false;
+ }
String transformName = partitionField.transform().toString();
if (!YEAR.equals(transformName)
&& !MONTH.equals(transformName)
@@ -332,7 +336,18 @@ public class IcebergUtils {
}
}
+ public static Expression convertToIcebergExpr(Expr expr, Schema schema, Set<Integer> unsafeNanosFields) {
+ // Drop the whole unsafe conjunct before conversion, including under NOT: negating a
+ // partially widened operand would incorrectly prune matching files.
+ return IcebergTimestampUtils.referencesUnsafe(expr, schema, unsafeNanosFields)
+ ? null : convertToIcebergExpr(expr, schema);
+ }
+
public static Expression convertToIcebergExpr(Expr expr, Schema schema) {
+ return convertToIcebergExpr(expr, schema, false);
+ }
+
+ private static Expression convertToIcebergExpr(Expr expr, Schema schema, boolean requireExact) {
if (expr == null) {
return null;
}
@@ -351,10 +366,12 @@ public class IcebergUtils {
CompoundPredicate compoundPredicate = (CompoundPredicate) expr;
switch (compoundPredicate.getOp()) {
case AND: {
- Expression left = convertToIcebergExpr(compoundPredicate.getChild(0), schema);
- Expression right = convertToIcebergExpr(compoundPredicate.getChild(1), schema);
+ Expression left = convertToIcebergExpr(compoundPredicate.getChild(0), schema, requireExact);
+ Expression right = convertToIcebergExpr(compoundPredicate.getChild(1), schema, requireExact);
if (left != null && right != null) {
expression = Expressions.and(left, right);
+ } else if (requireExact) {
+ return null;
} else if (left != null) {
return left;
} else if (right != null) {
@@ -363,15 +380,16 @@ public class IcebergUtils {
break;
}
case OR: {
- Expression left = convertToIcebergExpr(compoundPredicate.getChild(0), schema);
- Expression right = convertToIcebergExpr(compoundPredicate.getChild(1), schema);
+ Expression left = convertToIcebergExpr(compoundPredicate.getChild(0), schema, requireExact);
+ Expression right = convertToIcebergExpr(compoundPredicate.getChild(1), schema, requireExact);
if (left != null && right != null) {
expression = Expressions.or(left, right);
}
break;
}
case NOT: {
- Expression child = convertToIcebergExpr(compoundPredicate.getChild(0), schema);
+ // Negation must never invert a widened operand after a nanos conversion fails.
+ Expression child = convertToIcebergExpr(compoundPredicate.getChild(0), schema, true);
if (child != null) {
expression = Expressions.not(child);
}
@@ -391,11 +409,11 @@ public class IcebergUtils {
case LT:
case EQ_FOR_NULL:
BinaryPredicate eq = (BinaryPredicate) expr;
- SlotRef slotRef = convertDorisExprToSlotRef(eq.getChild(0));
+ SlotRef slotRef = convertDorisExprToSlotRef(eq.getChild(0), schema);
LiteralExpr literalExpr = null;
if (slotRef == null && eq.getChild(0).isLiteral()) {
literalExpr = (LiteralExpr) eq.getChild(0);
- slotRef = convertDorisExprToSlotRef(eq.getChild(1));
+ slotRef = convertDorisExprToSlotRef(eq.getChild(1), schema);
} else if (eq.getChild(1).isLiteral()) {
literalExpr = (LiteralExpr) eq.getChild(1);
}
@@ -447,7 +465,7 @@ public class IcebergUtils {
} else if (expr instanceof InPredicate) {
// InPredicate, only support a in (1,2,3)
InPredicate inExpr = (InPredicate) expr;
- SlotRef slotRef = convertDorisExprToSlotRef(inExpr.getChild(0));
+ SlotRef slotRef = convertDorisExprToSlotRef(inExpr.getChild(0), schema);
if (slotRef == null) {
return null;
}
@@ -478,7 +496,7 @@ public class IcebergUtils {
}
}
- return checkConversion(expression, schema);
+ return checkConversion(expression, schema, requireExact);
}
private static Types.NestedField getPushdownField(Schema schema, String colName) {
@@ -489,17 +507,19 @@ public class IcebergUtils {
return schema.caseInsensitiveFindField(colName);
}
- private static Expression checkConversion(Expression expression, Schema schema) {
+ private static Expression checkConversion(Expression expression, Schema schema, boolean requireExact) {
if (expression == null) {
return null;
}
switch (expression.op()) {
case AND: {
And andExpr = (And) expression;
- Expression left = checkConversion(andExpr.left(), schema);
- Expression right = checkConversion(andExpr.right(), schema);
+ Expression left = checkConversion(andExpr.left(), schema, requireExact);
+ Expression right = checkConversion(andExpr.right(), schema, requireExact);
if (left != null && right != null) {
return andExpr;
+ } else if (requireExact) {
+ return null;
} else if (left != null) {
return left;
} else if (right != null) {
@@ -510,8 +530,8 @@ public class IcebergUtils {
}
case OR: {
Or orExpr = (Or) expression;
- Expression left = checkConversion(orExpr.left(), schema);
- Expression right = checkConversion(orExpr.right(), schema);
+ Expression left = checkConversion(orExpr.left(), schema, requireExact);
+ Expression right = checkConversion(orExpr.right(), schema, requireExact);
if (left == null || right == null) {
return null;
} else {
@@ -520,7 +540,7 @@ public class IcebergUtils {
}
case NOT: {
Not notExpr = (Not) expression;
- Expression child = checkConversion(notExpr.child(), schema);
+ Expression child = checkConversion(notExpr.child(), schema, true);
if (child == null) {
return null;
} else {
@@ -545,6 +565,23 @@ public class IcebergUtils {
}
public static Object extractDorisLiteral(org.apache.iceberg.types.Type icebergType, Expr expr) {
+ if (icebergType.typeId() == TypeID.UUID) {
+ // UUID file bounds may use a different order from Iceberg's evaluator, even for equality.
+ // Keep value predicates as Doris residuals; null-count predicates are still safe.
+ return null;
+ }
+ if (icebergType.typeId() == TypeID.TIMESTAMP_NANO) {
+ if (expr instanceof NullLiteral || ((Types.TimestampNanoType) icebergType).shouldAdjustToUTC()) {
+ return null;
+ }
+ try {
+ // Iceberg treats raw Long literals as micros; preserve the unit explicitly.
+ return Expressions.nanos(IcebergTimestampUtils.parseNanos(expr.getStringValue(), ZoneOffset.UTC));
+ } catch (IllegalArgumentException | ArithmeticException | java.time.DateTimeException
+ | org.apache.doris.nereids.exceptions.AnalysisException e) {
+ return null;
+ }
+ }
TypeID icebergTypeID = icebergType.typeId();
if (expr instanceof BoolLiteral) {
BoolLiteral boolLiteral = (BoolLiteral) expr;
@@ -666,13 +703,19 @@ public class IcebergUtils {
return null;
}
- private static SlotRef convertDorisExprToSlotRef(Expr expr) {
+ private static SlotRef convertDorisExprToSlotRef(Expr expr, Schema schema) {
SlotRef slotRef = null;
if (expr instanceof SlotRef) {
slotRef = (SlotRef) expr;
} else if (expr instanceof CastExpr) {
if (expr.getChild(0) instanceof SlotRef) {
slotRef = (SlotRef) expr.getChild(0);
+ Types.NestedField field = getPushdownField(schema, slotRef.getColumnName());
+ // Rounding a nanos column changes which rows match; stripping that cast can prune matches.
+ if (field != null && field.type().typeId() == TypeID.TIMESTAMP_NANO
+ && !expr.getType().equals(Type.TIMESTAMP_NS)) {
+ return null;
+ }
}
}
return slotRef;
@@ -756,7 +799,7 @@ public class IcebergUtils {
case STRING:
return Type.STRING;
case UUID:
- return ScalarType.createVarbinaryType(16);
+ return Type.UUID;
case BINARY:
// Arbitrary binary payloads are not valid UTF-8 in general, so exposing them as
// STRING makes Arrow clients reject otherwise valid Iceberg values.
@@ -777,6 +820,10 @@ public class IcebergUtils {
return ScalarType.createTimeStampTzType(ICEBERG_DATETIME_SCALE_MS);
}
return ScalarType.createDatetimeV2Type(ICEBERG_DATETIME_SCALE_MS);
+ case TIMESTAMP_NANO:
+ // Unzoned nanos have a wall-clock carrier; zoned nanos remain unsupported.
+ return ((Types.TimestampNanoType) primitive).shouldAdjustToUTC()
+ ? Type.UNSUPPORTED : Type.TIMESTAMP_NS;
case TIME:
return Type.UNSUPPORTED;
default:
@@ -1183,6 +1230,8 @@ public class IcebergUtils {
long micros = (Long) value;
LocalTime time = LocalTime.ofNanoOfDay(micros * 1000);
return time.format(DateTimeFormatter.ISO_LOCAL_TIME);
+ case TIMESTAMP_NANO:
+ return value == null ? null : Transforms.identity(type).toHumanString(type, value);
case TIMESTAMP:
if (value == null) {
return null;
@@ -1328,10 +1377,13 @@ public class IcebergUtils {
} catch (NumberFormatException e) {
throw new IllegalArgumentException("Invalid Int string: " + value, e);
}
+ case TIMESTAMP_NANO:
+ // ADD COLUMN sends validated SQL text, while integer metadata is already nanos.
+ return Expressions.nanos(value.matches("[+-]?[0-9]+") ? Long.parseLong(value)
+ : IcebergTimestampUtils.parseNanos(value, ZoneOffset.UTC));
case LONG:
case TIME:
case TIMESTAMP:
- case TIMESTAMP_NANO:
try {
return Literal.of(Long.parseLong(value));
} catch (NumberFormatException e) {
@@ -1398,7 +1450,8 @@ public class IcebergUtils {
return valueStr;
case UUID:
if (!valueStr.startsWith("0x")) {
- return UUID.fromString(valueStr);
+ // Static SQL partitions accept the same compact UUID text as the sink cast.
+ return new org.apache.doris.nereids.trees.expressions.literal.UuidLiteral(valueStr).getValue();
}
ByteBuffer uuidBytes = decodeBinaryPartitionValue(valueStr);
Preconditions.checkArgument(uuidBytes.remaining() == 16, "UUID requires 16 bytes");
@@ -1423,6 +1476,9 @@ public class IcebergUtils {
case DATE:
// Parse date string (format: yyyy-MM-dd) to epoch day
return (int) LocalDate.parse(valueStr, DateTimeFormatter.ISO_LOCAL_DATE).toEpochDay();
+ case TIMESTAMP_NANO:
+ return IcebergTimestampUtils.parseNanos(valueStr,
+ org.apache.doris.nereids.util.DateUtils.getTimeZone());
case TIMESTAMP:
return parseTimestampToMicros(valueStr, (TimestampType) icebergType);
case TIME:
@@ -1627,7 +1683,7 @@ public class IcebergUtils {
return SingleValueParser.toJson(type, value);
}
String humanValue = Transforms.identity(type).toHumanString(type, value);
- if (type.typeId() == TypeID.TIMESTAMP) {
+ if (type.typeId() == TypeID.TIMESTAMP || type.typeId() == TypeID.TIMESTAMP_NANO) {
// Iceberg formats timestamps as ISO-8601 (for example 2024-01-01T00:00:00), while
// Doris' DATETIMEV2 default parser requires a space between the date and time.
String dorisValue = humanValue.replace('T', ' ');
@@ -1660,8 +1716,8 @@ public class IcebergUtils {
/**
* Return binary-like initial defaults in a lossless transport representation. These defaults
- * cannot be carried as raw Java strings and their Doris type is insufficient to identify them
- * when varbinary mapping is disabled, because UUID/BINARY/FIXED then map to STRING/CHAR.
+ * cannot be carried as raw Java strings. UUID defaults instead use canonical text and their
+ * native Doris UUID type.
*/
public static Map<Integer, String> getBase64EncodedInitialDefaults(Schema schema) {
Map<Integer, String> result = Maps.newHashMap();
@@ -1702,8 +1758,7 @@ public class IcebergUtils {
/**
* Return every binary-like field id, including list elements and map keys/values without a
* field-level default. BE needs this type identity while decoding binary values nested inside
- * a complex initial-default JSON value because Iceberg STRING, UUID, FIXED, and BINARY can all
- * map to a Doris string type.
+ * a complex initial-default JSON value, whose string tokens do not identify binary payloads.
*/
public static Set<Integer> getBinaryLikeFieldIds(Schema schema) {
return getBinaryLikeFieldIds(schema.columns());
@@ -1735,19 +1790,11 @@ public class IcebergUtils {
}
public static boolean isBinaryLike(org.apache.iceberg.types.Type type) {
- return type.typeId() == TypeID.UUID || type.typeId() == TypeID.BINARY
- || type.typeId() == TypeID.FIXED;
+ return type.typeId() == TypeID.BINARY || type.typeId() == TypeID.FIXED;
}
private static String serializeBinaryInitialDefault(org.apache.iceberg.types.Type type, Object value) {
- if (type.typeId() != TypeID.UUID) {
- return Transforms.identity(type).toHumanString(type, value);
- }
- UUID uuid = (UUID) value;
- ByteBuffer bytes = ByteBuffer.allocate(16);
- bytes.putLong(uuid.getMostSignificantBits());
- bytes.putLong(uuid.getLeastSignificantBits());
- return Base64.getEncoder().encodeToString(bytes.array());
+ return Transforms.identity(type).toHumanString(type, value);
}
/**
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUuidWriteConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUuidWriteConverter.java
new file mode 100644
index 00000000000..03ee58a9c7e
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUuidWriteConverter.java
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg;
+
+import org.apache.doris.nereids.trees.expressions.ArrayItemReference;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Array;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayMap;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.CreateMap;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.CreateNamedStruct;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ElementAt;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Lambda;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.MapLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StructLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.UuidLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.MapType;
+import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+import org.apache.doris.nereids.types.UuidType;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/** Validates UUID text before ordinary Iceberg sink coercion, including nested leaves. */
+public final class IcebergUuidWriteConverter {
+ private IcebergUuidWriteConverter() {
+ }
+
+ public static Expression convert(Expression input, DataType targetType) {
+ return TypeCoercionUtils.castIfNotSameType(convertUuid(input, targetType), targetType);
+ }
+
+ public static boolean containsUuid(DataType type) {
+ if (type instanceof ArrayType) {
+ return containsUuid(((ArrayType) type).getItemType());
+ }
+ if (type instanceof MapType) {
+ return containsUuid(((MapType) type).getKeyType()) || containsUuid(((MapType) type).getValueType());
+ }
+ if (type instanceof StructType) {
+ return ((StructType) type).getFields().stream().anyMatch(field -> containsUuid(field.getDataType()));
+ }
+ return type.isUuidType();
+ }
+
+ private static Expression convertUuid(Expression input, DataType targetType) {
+ if (!needsConversion(input.getDataType(), targetType)) {
+ return input;
+ }
+ // Parse serialized containers with UUID leaves kept as text. Nullable complex SerDes
+ // otherwise consume malformed UUIDs as NULL before the write validator can see them.
+ input = TypeCoercionUtils.castIfNotSameType(input,
+ uuidIntermediateType(input.getDataType(), targetType, true));
+ if (targetType instanceof ArrayType) {
+ DataType elementTarget = ((ArrayType) targetType).getItemType();
+ ArrayItemReference item = new ArrayItemReference("item", input);
+ return new ArrayMap(new Lambda(Collections.singletonList(item.getName()),
+ convertUuid(item.toSlot(), elementTarget), Collections.singletonList(item)));
+ }
+ if (targetType instanceof MapType) {
+ MapType target = (MapType) targetType;
+ if (input instanceof CreateMap || input instanceof MapLiteral) {
+ List<Expression> source = new ArrayList<>();
+ if (input instanceof MapLiteral) {
+ ((MapLiteral) input).getValue().forEach((key, value) -> {
+ source.add(key);
+ source.add(value);
+ });
+ } else {
+ source.addAll(input.children());
+ }
+ List<Expression> children = new ArrayList<>();
+ for (int i = 0; i < source.size(); i++) {
+ DataType targetChild = i % 2 == 0 ? target.getKeyType() : target.getValueType();
+ Expression child = source.get(i);
+ // A NULL-only map value has already been assigned TINYINT by generic function binding.
+ children.add(child instanceof NullLiteral ? new NullLiteral(targetChild)
+ : convertUuid(child, targetChild));
+ }
+ return TypeCoercionUtils.processBoundFunction(new CreateMap(children.toArray(new Expression[0])));
+ }
+ // Evaluate the map once, but change only UUID text leaves in the strict cast.
+ // Narrowing non-UUID siblings here would override ordinary sink cast semantics.
+ return new Cast(input, uuidIntermediateType(input.getDataType(), targetType, false), false, true);
+ }
+ if (targetType instanceof StructType) {
+ if (!(input instanceof SlotReference) && !(input instanceof StructLiteral)) {
+ // Bind the record once before null testing and field extraction; VALUES inputs may
+ // be volatile and have no source slot yet. The singleton array supplies that binding.
+ Expression singleton = TypeCoercionUtils.processBoundFunction(new Array(input));
+ Expression converted = convertUuid(singleton, ArrayType.of(targetType));
+ return TypeCoercionUtils.processBoundFunction(new ElementAt(converted, new IntegerLiteral(1)));
+ }
+ List<StructField> targetFields = ((StructType) targetType).getFields();
+ List<Expression> fields = new ArrayList<>();
+ for (int i = 0; i < targetFields.size(); i++) {
+ Expression field = input instanceof StructLiteral ? ((StructLiteral) input).getValue().get(i)
+ : TypeCoercionUtils.processBoundFunction(new ElementAt(input, new IntegerLiteral(i + 1)));
+ fields.add(new StringLiteral(targetFields.get(i).getName()));
+ fields.add(convertUuid(field, targetFields.get(i).getDataType()));
+ }
+ Expression result = TypeCoercionUtils.processBoundFunction(
+ new CreateNamedStruct(fields.toArray(new Expression[0])));
+ // Reconstructing a nullable struct must not turn NULL into a non-null struct of NULL fields.
+ return input.nullable() ? TypeCoercionUtils.processBoundFunction(
+ new If(new IsNull(input), new NullLiteral(result.getDataType()), result)) : result;
+ }
+ if (input instanceof StringLikeLiteral) {
+ return new UuidLiteral(((StringLikeLiteral) input).getStringValue());
+ }
+ // Reject malformed UUID text on write instead of silently replacing it with NULL.
+ return new Cast(input, targetType, false, true);
+ }
+
+ private static boolean needsConversion(DataType input, DataType semantic) {
+ if (input.isStringLikeType() && containsUuid(semantic)) {
+ return true;
+ }
+ if (semantic instanceof UuidType) {
+ return input.isStringLikeType();
+ }
+ if (semantic instanceof ArrayType && input instanceof ArrayType) {
+ return needsConversion(((ArrayType) input).getItemType(), ((ArrayType) semantic).getItemType());
+ }
+ if (semantic instanceof MapType && input instanceof MapType) {
+ return needsConversion(((MapType) input).getKeyType(), ((MapType) semantic).getKeyType())
+ || needsConversion(((MapType) input).getValueType(), ((MapType) semantic).getValueType());
+ }
+ if (semantic instanceof StructType && input instanceof StructType) {
+ List<StructField> inputs = ((StructType) input).getFields();
+ List<StructField> semantics = ((StructType) semantic).getFields();
+ if (inputs.size() != semantics.size()) {
+ // Ordinary sink coercion reports the incompatible struct arity.
+ return false;
+ }
+ for (int i = 0; i < inputs.size(); i++) {
+ if (needsConversion(inputs.get(i).getDataType(), semantics.get(i).getDataType())) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ /** Preserve source siblings while parsing containers or validating UUID text leaves. */
+ private static DataType uuidIntermediateType(DataType input, DataType target, boolean parseContainers) {
+ if (input.isStringLikeType() && containsUuid(target)) {
+ if (target.isUuidType()) {
+ return parseContainers ? input : target;
+ }
+ return parseContainers ? uuidTextType(target) : input;
+ }
+ if (input instanceof ArrayType && target instanceof ArrayType) {
+ return ArrayType.of(uuidIntermediateType(((ArrayType) input).getItemType(),
+ ((ArrayType) target).getItemType(), parseContainers));
+ }
+ if (input instanceof MapType && target instanceof MapType) {
+ return MapType.of(uuidIntermediateType(((MapType) input).getKeyType(),
+ ((MapType) target).getKeyType(), parseContainers),
+ uuidIntermediateType(((MapType) input).getValueType(),
+ ((MapType) target).getValueType(), parseContainers));
+ }
+ if (input instanceof StructType && target instanceof StructType) {
+ List<StructField> sourceFields = ((StructType) input).getFields();
+ List<StructField> targetFields = ((StructType) target).getFields();
+ if (sourceFields.size() == targetFields.size()) {
+ List<StructField> fields = new ArrayList<>();
+ for (int i = 0; i < sourceFields.size(); i++) {
+ StructField source = sourceFields.get(i);
+ fields.add(source.withDataType(uuidIntermediateType(source.getDataType(),
+ targetFields.get(i).getDataType(), parseContainers)));
+ }
+ return new StructType(fields);
+ }
+ }
+ return input;
+ }
+
+ private static DataType uuidTextType(DataType type) {
+ if (type.isUuidType()) {
+ return StringType.INSTANCE;
+ }
+ if (type instanceof ArrayType) {
+ return ArrayType.of(uuidTextType(((ArrayType) type).getItemType()));
+ }
+ if (type instanceof MapType) {
+ return MapType.of(uuidTextType(((MapType) type).getKeyType()),
+ uuidTextType(((MapType) type).getValueType()));
+ }
+ if (type instanceof StructType) {
+ List<StructField> fields = new ArrayList<>();
+ for (StructField field : ((StructType) type).getFields()) {
+ fields.add(field.withDataType(uuidTextType(field.getDataType())));
+ }
+ return new StructType(fields);
+ }
+ return type;
+ }
+
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContext.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContext.java
index 8b0aebb2e0b..837948dee8b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContext.java
@@ -607,13 +607,18 @@ public final class IcebergWriteSchemaContext {
case STRING:
return new StringLiteral((String) value);
case UUID:
- return binaryExpression(uuidBytes((UUID) value), targetType);
+ return new org.apache.doris.nereids.trees.expressions.literal.UuidLiteral(value.toString());
case FIXED:
case BINARY:
return binaryExpression(byteBufferBytes((ByteBuffer) value), targetType);
case DATE:
LocalDate date = LocalDate.ofEpochDay(((Integer) value).longValue());
return new DateV2Literal(date.getYear(), date.getMonthValue(), date.getDayOfMonth());
+ case TIMESTAMP_NANO:
+ // Identity formatting retains all nine fractional digits, including negative epochs.
+ return new org.apache.doris.nereids.trees.expressions.literal.TimeStampNsLiteral(
+ org.apache.iceberg.transforms.Transforms.identity(icebergType)
+ .toHumanString(icebergType, value).replace('T', ' '));
case TIMESTAMP:
long micros = (Long) value;
Types.TimestampType timestampType = (Types.TimestampType) icebergType;
@@ -662,7 +667,7 @@ public final class IcebergWriteSchemaContext {
.map(Literal.class::cast).collect(ImmutableList.toImmutableList());
return new ArrayLiteral(literalItems, targetType);
}
- // Legacy UUID/FIXED/BINARY mapping uses UNHEX to materialize raw bytes. Container
+ // Legacy FIXED/BINARY mapping uses UNHEX to materialize raw bytes. Container
// literals accept literal children only, so preserve that expression in the existing
// array function path; UNHEX remains executable by older BEs during a rolling upgrade.
return TypeCoercionUtils.castIfNotSameType(new Array(items), targetType);
@@ -727,13 +732,6 @@ public final class IcebergWriteSchemaContext {
new CreateNamedStruct(namedFields.toArray(new Expression[0])), targetType);
}
- private static byte[] uuidBytes(UUID value) {
- return ByteBuffer.allocate(16)
- .putLong(value.getMostSignificantBits())
- .putLong(value.getLeastSignificantBits())
- .array();
- }
-
private static Expression binaryExpression(byte[] bytes, DataType targetType) {
if (targetType instanceof VarBinaryType) {
return new VarBinaryLiteral(targetType, bytes);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlanner.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlanner.java
index 42a969f5625..e7ce645f588 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlanner.java
@@ -93,6 +93,14 @@ public class RewriteDataFilePlanner {
// Apply WHERE condition if specified
if (parameters.hasWhereCondition()) {
+ // Rewrite selects whole files, so unsafe nanos pruning must be rejected, not widened.
+ if (org.apache.doris.datasource.iceberg.IcebergTimestampUtils.referencesUnsafe(
+ parameters.getWhereCondition().get(), icebergTable.schema(),
+ org.apache.doris.datasource.iceberg.IcebergTimestampUtils.unsafeNanosFields(
+ icebergTable, icebergTable.schema()))) {
+ throw new org.apache.doris.common.UserException(
+ "WHERE condition for rewrite_data_files cannot prune historical DATE bounds");
+ }
org.apache.iceberg.expressions.Expression icebergExpression = IcebergNereidsUtils
.convertNereidsToIcebergExpression(parameters.getWhereCondition().get(), icebergTable.schema());
tableScan = tableScan.filter(icebergExpression);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java
index f8a7f00a23d..06fb012c1ec 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java
@@ -53,6 +53,7 @@ import org.apache.doris.datasource.iceberg.IcebergRuntimeContext;
import org.apache.doris.datasource.iceberg.IcebergSnapshot;
import org.apache.doris.datasource.iceberg.IcebergSnapshotCacheValue;
import org.apache.doris.datasource.iceberg.IcebergSysExternalTable;
+import org.apache.doris.datasource.iceberg.IcebergTimestampUtils;
import org.apache.doris.datasource.iceberg.IcebergUtils;
import org.apache.doris.datasource.iceberg.cache.IcebergManifestCacheLoader;
import org.apache.doris.datasource.iceberg.cache.ManifestCacheValue;
@@ -1916,13 +1917,14 @@ public class IcebergScanNode extends FileQueryScanNode {
scan = DorisDataTableScan.wrap(scan);
}
Schema scanSchema = scan.schema();
+ Set<Integer> unsafeNanosFields = IcebergTimestampUtils.unsafeNanosFields(icebergTable, scanSchema);
// set filter
List<Expression> expressions = new ArrayList<>();
for (Expr conjunct : conjuncts) {
// Ref/snapshot selection can change field IDs and names, so every pruning expression
// must be bound against the schema selected by this scan rather than the current table.
- Expression expression = IcebergUtils.convertToIcebergExpr(conjunct, scanSchema);
+ Expression expression = IcebergUtils.convertToIcebergExpr(conjunct, scanSchema, unsafeNanosFields);
if (expression != null) {
expressions.add(expression);
}
@@ -2407,11 +2409,12 @@ public class IcebergScanNode extends FileQueryScanNode {
}
ExternalTable targetExternalTable = (ExternalTable) source.getTargetTable();
Schema scanSchema = scan.schema();
+ Set<Integer> unsafeNanosFields = IcebergTimestampUtils.unsafeNanosFields(icebergTable, scanSchema);
// Convert query conjuncts to Iceberg filter expression
// This combines all predicates with AND logic for partition/file pruning
Expression filterExpr = conjuncts.stream()
- .map(conjunct -> IcebergUtils.convertToIcebergExpr(conjunct, scanSchema))
+ .map(conjunct -> IcebergUtils.convertToIcebergExpr(conjunct, scanSchema, unsafeNanosFields))
.filter(Objects::nonNull)
.reduce(Expressions.alwaysTrue(), Expressions::and);
@@ -2801,6 +2804,7 @@ public class IcebergScanNode extends FileQueryScanNode {
case UUID:
case DATE:
case TIME:
+ case TIMESTAMP_NANO:
case TIMESTAMP:
return partitionValue;
default:
@@ -2926,10 +2930,11 @@ public class IcebergScanNode extends FileQueryScanNode {
}
}
Schema scanSchema = scan.schema();
+ Set<Integer> unsafeNanosFields = IcebergTimestampUtils.unsafeNanosFields(icebergTable, scanSchema);
List<Expression> expressions = new ArrayList<>();
for (Expr conjunct : conjuncts) {
- Expression expression = IcebergUtils.convertToIcebergExpr(conjunct, scanSchema);
+ Expression expression = IcebergUtils.convertToIcebergExpr(conjunct, scanSchema, unsafeNanosFields);
if (expression != null) {
expressions.add(expression);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java
index 946e69b5feb..5bcf9fcadd7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java
@@ -118,6 +118,9 @@ public class JdbcPostgreSQLClient extends JdbcClient {
protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) {
String pgType = fieldSchema.getDataTypeName().orElse("unknown");
switch (pgType) {
+ // Preserve the external logical UUID independently of binary mapping options.
+ case "uuid":
+ return Type.UUID;
case "int2":
case "smallserial":
return Type.SMALLINT;
@@ -184,7 +187,6 @@ public class JdbcPostgreSQLClient extends JdbcClient {
case "macaddr":
case "macaddr8":
case "varbit":
- case "uuid":
case "xml":
case "hstore":
case "json":
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java
index e3300d8ce5b..788385019e9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcSQLServerClient.java
@@ -89,6 +89,9 @@ public class JdbcSQLServerClient extends JdbcClient {
String sqlserverType = identityBaseType(originSqlserverType, fieldSchema.getDataType());
switch (sqlserverType) {
+ // Preserve the external logical UUID independently of binary mapping options.
+ case "uniqueidentifier":
+ return Type.UUID;
case "bit":
return Type.BOOLEAN;
case "tinyint":
@@ -134,7 +137,6 @@ public class JdbcSQLServerClient extends JdbcClient {
case "text":
case "ntext":
case "time":
- case "uniqueidentifier":
case "timestamp":
return ScalarType.createStringType();
case "image":
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java
index 7551a1bab4b..4869ec20f80 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcTrinoClient.java
@@ -33,6 +33,8 @@ public class JdbcTrinoClient extends JdbcClient {
protected Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) {
String trinoType = fieldSchema.getDataTypeName().orElse("unknown");
switch (trinoType) {
+ case "uuid":
+ return Type.UUID;
case "integer":
return Type.INT;
case "bigint":
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
index d75a2b8aa50..438edf3ae7f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java
@@ -522,7 +522,8 @@ public class JdbcScanNode extends ExternalScanNode {
}
private static boolean shouldPushDownConjunct(TOdbcTableType tableType, Expr expr) {
- if (tableType == TOdbcTableType.CLICKHOUSE
+ // SQL Server GUID ordering, like ClickHouse UUID ordering, differs from Doris unsigned order.
+ if ((tableType == TOdbcTableType.CLICKHOUSE || tableType == TOdbcTableType.SQLSERVER)
&& expr.contains((Expr child) -> child.getType().isUuid())) {
if (expr instanceof BinaryPredicate) {
BinaryPredicate.Operator op = ((BinaryPredicate) expr).getOp();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java
index 464a1c02ea0..dec21e8476f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/DorisToPaimonTypeVisitor.java
@@ -114,6 +114,8 @@ public class DorisToPaimonTypeVisitor extends DorisTypeVisitor<DataType> {
// Paimon interprets numeric timestamp strings in units derived from this precision.
return new TimestampType(primitiveType.equals(PrimitiveType.DATETIMEV2)
? ((ScalarType) atomic).getScalarScale() : 0);
+ } else if (primitiveType.equals(PrimitiveType.TIMESTAMP_NS)) {
+ return new TimestampType(9);
} else if (primitiveType.equals(PrimitiveType.TIMESTAMPTZ)) {
// TIMESTAMPTZ represents an instant, so map it to Paimon's local-zoned timestamp.
return new LocalZonedTimestampType(((ScalarType) atomic).getScalarScale());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
index 2a362ebebfd..8a3d8330203 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
@@ -345,7 +345,8 @@ public class PaimonUtil {
if (dataType instanceof org.apache.paimon.types.TimestampType) {
tsScale = ((org.apache.paimon.types.TimestampType) dataType).getPrecision();
if (tsScale > 6) {
- tsScale = 6;
+ // Preserve sub-microsecond digits for unzoned external timestamps.
+ return Type.TIMESTAMP_NS;
}
} else if (dataType instanceof org.apache.paimon.types.LocalZonedTimestampType) {
tsScale = ((org.apache.paimon.types.LocalZonedTimestampType) dataType).getPrecision();
@@ -398,6 +399,21 @@ public class PaimonUtil {
return paimonPrimitiveTypeToDorisType(type, enableVarbinaryMapping, enableTimestampTzMapping);
}
+ public static boolean containsTimestampNs(Type type) {
+ if (type.isTimeStampNs()) {
+ return true;
+ } else if (type.isArrayType()) {
+ return containsTimestampNs(((org.apache.doris.catalog.ArrayType) type).getItemType());
+ } else if (type.isMapType()) {
+ org.apache.doris.catalog.MapType mapType = (org.apache.doris.catalog.MapType) type;
+ return containsTimestampNs(mapType.getKeyType()) || containsTimestampNs(mapType.getValueType());
+ } else if (type.isStructType()) {
+ return ((org.apache.doris.catalog.StructType) type).getFields().stream()
+ .anyMatch(field -> containsTimestampNs(field.getType()));
+ }
+ return false;
+ }
+
public static boolean containsVariant(Type type) {
if (type.isVariantType()) {
return true;
@@ -838,6 +854,11 @@ public class PaimonUtil {
// Format through Doris' target type instead of translating between Paimon's
// timestamp text and Doris' partition-literal syntax by hand.
TimestampType timestampType = (TimestampType) type;
+ if (timestampType.getPrecision() > 6) {
+ // Partition metadata must preserve the same precision as the mapped column.
+ return new org.apache.doris.analysis.TimeStampNsLiteral(
+ ((Timestamp) value).toLocalDateTime()).getStringValue();
+ }
ScalarType dorisType = ScalarType.createDatetimeV2Type(
Math.min(timestampType.getPrecision(), 6));
return new DateLiteral(((Timestamp) value).toLocalDateTime(), dorisType)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
index 111bbc905c0..425f5abaee3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
@@ -1025,7 +1025,7 @@ public class PaimonScanNode extends FileQueryScanNode {
&& !isIncremental && providerModeTranslatable && !queryAuthTable
&& !restTokenTable
&& !dvMergeOnRead && !splitDvNotMaterialized
- && !orcLtzSchema && !projectedVariant && !externalFileSplit
+ && !orcLtzSchema && !projectedVariant && !needsNanosecondReader() && !externalFileSplit
&& !deduplicateIgnoreDelete && !rustUnsupportedMergeOption
&& schemeCapabilityVerified && hdfsBackendVerified
&& paimonFileStoreTable != null;
@@ -1718,8 +1718,16 @@ public class PaimonScanNode extends FileQueryScanNode {
return selectFeSplitSizeForBe(fallbackSize, thriftFormat, true);
}
+ private boolean needsNanosecondReader() {
+ // Slots include filter-only columns. JNI retains INT96/ORC nanos, including nested values.
+ return desc.getSlots().stream().anyMatch(slot -> PaimonUtil.containsTimestampNs(slot.getType()));
+ }
+
@VisibleForTesting
public boolean supportNativeReader(Optional<List<RawFile>> optRawFiles) {
+ if (needsNanosecondReader()) {
+ return false;
+ }
if (!optRawFiles.isPresent()) {
return false;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java
index d490474489d..a3b76e74198 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonValueConverter.java
@@ -144,6 +144,11 @@ public class PaimonValueConverter extends DataTypeDefaultVisitor<Object> {
}
public Timestamp visit(TimestampType timestampType) {
+ if (expr instanceof org.apache.doris.analysis.TimeStampNsLiteral) {
+ // A nanos predicate must not be narrowed before Paimon evaluates it.
+ return Timestamp.fromLocalDateTime(
+ ((org.apache.doris.analysis.TimeStampNsLiteral) expr).toLocalDateTime());
+ }
if (expr instanceof DateLiteral) {
DateLiteral dateLiteral = (DateLiteral) expr;
Calendar instance = Calendar.getInstance(TimeZone.getTimeZone("GMT"));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java
index cb56f0fdd24..243f78afd50 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/trinoconnector/TrinoConnectorExternalTable.java
@@ -180,6 +180,8 @@ public class TrinoConnectorExternalTable extends ExternalTable {
return Type.DOUBLE;
} else if (type instanceof CharType) {
return Type.CHAR;
+ } else if (type instanceof io.trino.spi.type.UuidType) {
+ return Type.UUID;
} else if (type instanceof VarcharType) {
return Type.STRING;
// } else if (type instanceof BinaryType) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
index b91c2f9864c..f4669ec001b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
@@ -40,6 +40,7 @@ import org.apache.doris.datasource.iceberg.IcebergExternalTable;
import org.apache.doris.datasource.iceberg.IcebergMvccSnapshot;
import org.apache.doris.datasource.iceberg.IcebergSnapshotCacheValue;
import org.apache.doris.datasource.iceberg.IcebergUtils;
+import org.apache.doris.datasource.iceberg.IcebergUuidWriteConverter;
import org.apache.doris.datasource.iceberg.IcebergVariantWriteAnalyzer;
import org.apache.doris.datasource.iceberg.IcebergWriteSchemaContext;
import org.apache.doris.datasource.jdbc.JdbcExternalDatabase;
@@ -452,9 +453,11 @@ public class BindSink implements AnalysisRuleFactory {
// do not process explicitly use DEFAULT value here:
// insert into table t values(DEFAULT)
&& !(columnToChildOutput.get(column) instanceof DefaultValueSlot)) {
- Alias output = new Alias(TypeCoercionUtils.castIfNotSameType(
- columnToChildOutput.get(column), DataType.fromCatalogType(column.getType())),
- column.getName());
+ Expression input = columnToChildOutput.get(column);
+ DataType targetType = DataType.fromCatalogType(column.getType());
+ Alias output = new Alias(icebergWriteSchemaContext.isPresent()
+ ? IcebergUuidWriteConverter.convert(input, targetType)
+ : TypeCoercionUtils.castIfNotSameType(input, targetType), column.getName());
columnToOutput.put(column.getName(), output);
columnToReplaced.put(column.getName(), output.toSlot());
replaceMap.put(output.toSlot(), output.child());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/TimeStampNsLiteral.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/TimeStampNsLiteral.java
index 9e36fb6813d..95784c93bda 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/TimeStampNsLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/TimeStampNsLiteral.java
@@ -54,7 +54,7 @@ public final class TimeStampNsLiteral extends DateLiteral {
private final long nanosecond;
public TimeStampNsLiteral(String value) {
- this(parse(value));
+ this(parseDateTime(value, DateUtils.getTimeZone()));
}
/** Construct a TIMESTAMP_NS literal from civil datetime fields. */
@@ -76,7 +76,8 @@ public final class TimeStampNsLiteral extends DateLiteral {
value.getHour(), value.getMinute(), value.getSecond(), value.getNano());
}
- private static LocalDateTime parse(String value) {
+ /** Parse civil fields using Doris literal rules and an explicit destination timezone. */
+ public static LocalDateTime parseDateTime(String value, ZoneId targetZone) {
TemporalAccessor temporal = parseDateTime(value, DateUtils.NANOSECOND_SCALE + 1).get();
long year = DateUtils.getOrDefault(temporal, ChronoField.YEAR);
long month = DateUtils.getOrDefault(temporal, ChronoField.MONTH_OF_YEAR);
@@ -95,7 +96,7 @@ public final class TimeStampNsLiteral extends DateLiteral {
ZoneId zoneId = temporal.query(TemporalQueries.zone());
if (zoneId != null) {
Instant instant = DateTimeLiteral.convertLocalToInstantPreservingFraction(result, zoneId);
- result = LocalDateTime.ofInstant(instant, DateUtils.getTimeZone());
+ result = LocalDateTime.ofInstant(instant, targetZone);
}
return result;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java
index e3b93c5656a..fea4c9c8d67 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java
@@ -26,6 +26,7 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.common.FeNameFormat;
import org.apache.doris.common.util.SqlUtils;
import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.literal.TimeStampNsLiteral;
import org.apache.doris.nereids.types.ArrayType;
import org.apache.doris.nereids.types.BigIntType;
import org.apache.doris.nereids.types.BitmapType;
@@ -509,6 +510,13 @@ public class ColumnDefinition {
try {
ColumnDef.validateDefaultValue(type.toCatalogDataType(),
defaultValue.get().getValue(), defaultValue.get().getDefaultValueExprDef());
+ if (!isOlap && type.isTimeStampNsType()
+ && defaultValue.get().getDefaultValueExprDef() == null) {
+ // Connectors receive civil SQL defaults, not numeric epoch carriers. Normalize
+ // with the same parser as validation so compact dates cannot become epoch nanos.
+ defaultValue = Optional.of(new DefaultValue(
+ new TimeStampNsLiteral(defaultValue.get().getValue()).getStringValue()));
+ }
} catch (Exception e) {
throw new AnalysisException(e.getMessage(), e);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
index 6b6c484fcb3..907095dcdd6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
@@ -30,6 +30,7 @@ import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.datasource.iceberg.IcebergExternalTable;
import org.apache.doris.datasource.iceberg.IcebergUtils;
+import org.apache.doris.datasource.iceberg.IcebergUuidWriteConverter;
import org.apache.doris.datasource.iceberg.IcebergVariantWriteAnalyzer;
import org.apache.doris.datasource.iceberg.IcebergWriteSchemaContext;
import org.apache.doris.datasource.jdbc.JdbcExternalTable;
@@ -527,7 +528,7 @@ public class InsertUtils {
NamedExpression defaultExpression = generateDefaultExpression(
column, icebergWriteSchemaContext);
addColumnValue(analyzer, optimizedRowConstructor, defaultExpression,
- null, rewriteContext, strictCast);
+ null, rewriteContext, strictCast, isIcebergSink);
}
} else {
if (CollectionUtils.isNotEmpty(unboundLogicalSink.getColNames())) {
@@ -567,7 +568,7 @@ public class InsertUtils {
NamedExpression defaultExpression = generateDefaultExpression(
sameNameColumn, icebergWriteSchemaContext);
addColumnValue(analyzer, optimizedRowConstructor, defaultExpression,
- null, rewriteContext, strictCast);
+ null, rewriteContext, strictCast, isIcebergSink);
} else {
NamedExpression value = resolveIcebergDefaultReferences(
values.get(i), icebergWriteSchemaContext,
@@ -575,7 +576,7 @@ public class InsertUtils {
DataType targetType = targetTypeForInlineValue(
sameNameColumn, value, isPaimonSink, isIcebergSink);
addColumnValue(analyzer, optimizedRowConstructor, value,
- targetType, rewriteContext, strictCast);
+ targetType, rewriteContext, strictCast, isIcebergSink);
}
}
} else {
@@ -593,7 +594,7 @@ public class InsertUtils {
NamedExpression defaultExpression = generateDefaultExpression(
columns.get(i), icebergWriteSchemaContext);
addColumnValue(analyzer, optimizedRowConstructor, defaultExpression,
- null, rewriteContext, strictCast);
+ null, rewriteContext, strictCast, isIcebergSink);
} else {
NamedExpression value = resolveIcebergDefaultReferences(
values.get(i), icebergWriteSchemaContext,
@@ -601,7 +602,7 @@ public class InsertUtils {
DataType targetType = targetTypeForInlineValue(
columns.get(i), value, isPaimonSink, isIcebergSink);
addColumnValue(analyzer, optimizedRowConstructor, value, targetType,
- rewriteContext, strictCast);
+ rewriteContext, strictCast, isIcebergSink);
}
}
}
@@ -752,7 +753,18 @@ public class InsertUtils {
Optional<ExpressionAnalyzer> analyzer,
ImmutableList.Builder<NamedExpression> optimizedRowConstructor,
NamedExpression value, DataType targetType, ExpressionRewriteContext rewriteContext,
- boolean strictCast) {
+ boolean strictCast, boolean isIcebergSink) {
+ if (isIcebergSink && targetType != null && IcebergUuidWriteConverter.containsUuid(targetType)) {
+ // VALUES normalization runs before BindSink; keep UUID text intact until it can be validated.
+ if (!analyzer.isPresent()) {
+ optimizedRowConstructor.add(value);
+ return;
+ }
+ value = (NamedExpression) analyzer.get().analyze(value,
+ new ExpressionRewriteContext(analyzer.get().getCascadesContext()));
+ Expression input = value instanceof Alias ? value.child(0) : value;
+ value = new Alias(IcebergUuidWriteConverter.convert(input, targetType), value.getName());
+ }
if (targetType != null) {
// In strict cast/insert mode, we don't cast to target varchar type here,
// we cast to varchar max here and do substring accordingly in BindSink.
diff --git a/fe/fe-core/src/main/java/org/apache/iceberg/DeleteFileIndex.java b/fe/fe-core/src/main/java/org/apache/iceberg/DeleteFileIndex.java
index da7f7803835..78aae9240f4 100644
--- a/fe/fe-core/src/main/java/org/apache/iceberg/DeleteFileIndex.java
+++ b/fe/fe-core/src/main/java/org/apache/iceberg/DeleteFileIndex.java
@@ -290,6 +290,9 @@ public class DeleteFileIndex {
T deleteLower,
T deleteUpper) {
Type.PrimitiveType type = field.type().asPrimitiveType();
+ if (isHistoricalDateBound(type, dataLowerBuf) || isHistoricalDateBound(type, dataUpperBuf)) {
+ return true;
+ }
Comparator<T> comparator = Comparators.forType(type);
T dataLower = Conversions.fromByteBuffer(type, dataLowerBuf);
@@ -305,6 +308,13 @@ public class DeleteFileIndex {
return true;
}
+ private static boolean isHistoricalDateBound(Type type, ByteBuffer bound) {
+ // DATE-to-nanos promotion preserves four-byte day bounds in both data and delete manifests.
+ // Widening them to long does not convert their units, even after the key leaves the scan schema.
+ return type.typeId() == Type.TypeID.TIMESTAMP_NANO
+ && bound != null && bound.remaining() == Integer.BYTES;
+ }
+
private static boolean allNonNull(Map<Integer, Long> nullValueCounts, Types.NestedField field) {
if (field.isRequired()) {
return true;
@@ -929,7 +939,7 @@ public class DeleteFileIndex {
Type type = field.type();
if (type.isPrimitiveType()) {
ByteBuffer bound = bounds.get(id);
- if (bound != null) {
+ if (bound != null && !isHistoricalDateBound(type, bound)) {
converted.put(id, Conversions.fromByteBuffer(type, bound));
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/UuidLiteralTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/UuidLiteralTest.java
index 7fa3019583d..33cac9e3695 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/UuidLiteralTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/UuidLiteralTest.java
@@ -56,4 +56,28 @@ public class UuidLiteralTest {
JdbcScanNode.conjunctExprToString(TOdbcTableType.CLICKHOUSE, predicate, null));
}
}
+
+ @Test
+ public void testTrinoAndPrestoUseTypedUuidLiterals() throws Exception {
+ org.apache.doris.catalog.JdbcTable table = org.mockito.Mockito.mock(org.apache.doris.catalog.JdbcTable.class);
+ UuidLiteral value = new UuidLiteral("80000000000000000000000000000000");
+ SlotRef slot = new SlotRef(null, "u");
+ slot.setType(Type.UUID);
+ for (TOdbcTableType dialect : Arrays.asList(TOdbcTableType.TRINO, TOdbcTableType.PRESTO)) {
+ org.mockito.Mockito.when(table.getJdbcTableType()).thenReturn(dialect);
+ org.mockito.Mockito.when(table.getProperRemoteColumnName(dialect, "u")).thenReturn("u");
+ Assert.assertEquals("UUID '80000000-0000-0000-0000-000000000000'",
+ value.toExternalSql(TableType.JDBC_EXTERNAL_TABLE, table));
+ Assert.assertEquals("(u = UUID '80000000-0000-0000-0000-000000000000')",
+ JdbcScanNode.conjunctExprToString(dialect,
+ new BinaryPredicate(BinaryPredicate.Operator.EQ, slot, value), table));
+ for (boolean notIn : Arrays.asList(false, true)) {
+ Assert.assertEquals("u" + (notIn ? " NOT" : "")
+ + " IN (UUID '80000000-0000-0000-0000-000000000000')",
+ JdbcScanNode.conjunctExprToString(dialect,
+ new InPredicate(slot, Arrays.asList(value), notIn), table));
+ }
+ }
+ }
+
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalTimestampMappingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalTimestampMappingTest.java
index 42e0c2dfdbd..181bb1e1c6c 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalTimestampMappingTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalTimestampMappingTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.datasource;
import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.StructType;
+import org.apache.doris.catalog.Type;
import org.apache.doris.datasource.hive.HiveMetaStoreClientHelper;
import org.apache.doris.datasource.iceberg.IcebergUtils;
import org.apache.doris.datasource.paimon.PaimonUtil;
@@ -106,8 +107,9 @@ public class ExternalTimestampMappingTest {
@Test
public void testPaimonTimestampPrecisionAndSemanticsIgnoreLegacyFlag() {
for (boolean legacyFlag : new boolean[] {false, true}) {
- for (int precision : new int[] {0, 3, 6, 9}) {
- Assert.assertEquals(ScalarType.createDatetimeV2Type(Math.min(precision, 6)),
+ for (int precision : new int[] {0, 3, 6, 7, 8, 9}) {
+ // Unzoned precision above six retains nanos regardless of the legacy flag.
+ Assert.assertEquals(precision > 6 ? Type.TIMESTAMP_NS : ScalarType.createDatetimeV2Type(precision),
PaimonUtil.paimonTypeToDorisType(new TimestampType(precision), false, legacyFlag));
Assert.assertEquals(ScalarType.createTimeStampTzType(Math.min(precision, 6)),
PaimonUtil.paimonTypeToDorisType(new LocalZonedTimestampType(precision), false, legacyFlag));
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/HudiUuidMappingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/HudiUuidMappingTest.java
new file mode 100644
index 00000000000..8f9f4c5925a
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/HudiUuidMappingTest.java
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.hudi;
+
+import org.apache.doris.catalog.ArrayType;
+import org.apache.doris.catalog.MapType;
+import org.apache.doris.catalog.StructType;
+import org.apache.doris.catalog.Type;
+
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+
+public class HudiUuidMappingTest {
+ @Test
+ public void testUuidSurvivesCatalogSchemaRoundTrip() {
+ Schema uuid = LogicalTypes.uuid().addToSchema(Schema.create(Schema.Type.STRING));
+ Schema nullableUuid = Schema.createUnion(Schema.create(Schema.Type.NULL), uuid);
+ Schema record = Schema.createRecord("uuid_record", null, null, false);
+ record.setFields(Arrays.asList(new Schema.Field("id", nullableUuid),
+ new Schema.Field("ids", Schema.createArray(nullableUuid)),
+ new Schema.Field("by_name", Schema.createMap(nullableUuid))));
+ Schema normalized = AvroInternalSchemaConverter.convert(
+ AvroInternalSchemaConverter.convert(record), "uuid_record");
+ Assert.assertEquals(Schema.Type.FIXED, normalized.getField("id").schema().getTypes().get(1).getType());
+ for (Schema schema : Arrays.asList(record, normalized)) {
+ StructType mapped = (StructType) HudiUtils.fromAvroHudiTypeToDorisType(schema);
+ Assert.assertEquals(Type.UUID, mapped.getFields().get(0).getType());
+ Assert.assertEquals(ArrayType.create(Type.UUID, true), mapped.getFields().get(1).getType());
+ Assert.assertEquals(new MapType(Type.STRING, Type.UUID), mapped.getFields().get(2).getType());
+ }
+ }
+
+ @Test
+ public void testNormalizedUuidUsesBinaryHiveCarrier() {
+ Schema uuid = LogicalTypes.uuid().addToSchema(Schema.create(Schema.Type.STRING));
+ Schema record = Schema.createRecord("uuid_record", null, null, false);
+ record.setFields(Arrays.asList(new Schema.Field("id", uuid),
+ new Schema.Field("ids", Schema.createArray(uuid)),
+ new Schema.Field("by_name", Schema.createMap(uuid))));
+ Schema normalized = AvroInternalSchemaConverter.convert(
+ AvroInternalSchemaConverter.convert(record), "uuid_record");
+ Assert.assertEquals("struct<id:binary,ids:array<binary>,by_name:map<string,binary>>",
+ HudiUtils.convertAvroToHiveType(normalized));
+ Assert.assertEquals("string", HudiUtils.convertAvroToHiveType(uuid));
+ }
+
+ @Test
+ public void testNonUuidMappingsArePreserved() {
+ for (Schema schema : Arrays.asList(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.BYTES),
+ Schema.createFixed("ordinary_fixed", null, null, 16))) {
+ Assert.assertEquals(Type.STRING, HudiUtils.fromAvroHudiTypeToDorisType(schema));
+ Assert.assertEquals("string", HudiUtils.convertAvroToHiveType(schema));
+ }
+ Schema decimal = LogicalTypes.decimal(10, 2).addToSchema(Schema.createFixed("amount", null, null, 8));
+ Assert.assertEquals("decimal(10,2)", HudiUtils.convertAvroToHiveType(decimal));
+ Assert.assertTrue(HudiUtils.fromAvroHudiTypeToDorisType(decimal).isDecimalV3());
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergDDLAndDMLPlanTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergDDLAndDMLPlanTest.java
index aa29e58f31a..b8de1d7cda1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergDDLAndDMLPlanTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergDDLAndDMLPlanTest.java
@@ -234,6 +234,8 @@ public class IcebergDDLAndDMLPlanTest extends TestWithFeService {
Mockito.doReturn(mockedTableScan).when(mockedTableScan).useRef(ArgumentMatchers.any());
Mockito.doReturn(mockedTableScan).when(mockedTableScan).filter(ArgumentMatchers.<org.apache.iceberg.expressions.Expression>any());
Mockito.doReturn(mockedTableScan).when(mockedTableScan).planWith(ArgumentMatchers.any());
+ // Projection must retain a real schema; deep stubs manufacture an empty Schema mock.
+ Mockito.doReturn(mockedTableScan).when(mockedTableScan).project(ArgumentMatchers.any(Schema.class));
Mockito.doReturn(null).when(mockedTableScan).snapshot();
// Keep the scan schema aligned with the mocked table schema. IcebergScanNode reads the
// selected scan schema when serializing initial defaults, and several tests temporarily
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
index b2959868a29..d6b4038496a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
@@ -32,6 +32,7 @@ import org.apache.iceberg.PartitionField;
import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Schema;
import org.apache.iceberg.transforms.Transform;
+import org.apache.iceberg.types.Types;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -124,6 +125,9 @@ public class IcebergExternalTableTest {
Assertions.assertFalse(spyTable.isValidRelatedTableCached());
Mockito.when(icebergTable.schema()).thenReturn(schema);
+ // Partition eligibility inspects the source type in the spec's own schema.
+ Mockito.when(spec.schema()).thenReturn(schema);
+ Mockito.when(schema.findType(1)).thenReturn(Types.TimestampType.withoutZone());
Mockito.when(schema.findColumnName(ArgumentMatchers.anyInt())).thenReturn("col1");
Mockito.doReturn(mockTransform("hour")).when(field).transform();
Mockito.when(field.sourceId()).thenReturn(1);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java
index 2b4b390462a..6d1b0b06a40 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java
@@ -705,7 +705,7 @@ public class IcebergMetadataOpsValidationTest {
mockWritableTableLease(mockedIcebergUtils, dorisTable, icebergTable);
ops.modifyColumn(dorisTable, ColumnPath.of("top_uuid"),
- new Column("top_uuid", ScalarType.createVarbinaryType(16), true), null, 1L);
+ new Column("top_uuid", Type.UUID, true), null, 1L);
ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.tz_value"),
new Column("tz_value", ScalarType.createTimeStampTzType(6), true), null, 1L);
}
@@ -1678,15 +1678,15 @@ public class IcebergMetadataOpsValidationTest {
}
private StructType mappedPayloadDorisType(Type metricType, int fixedLength, Type mapKeyType) {
+ // UUID annotations retain their logical type even when binary mapping is enabled.
return new StructType(
- new StructField("uuid_value", ScalarType.createVarbinaryType(16)),
+ new StructField("uuid_value", Type.UUID),
new StructField("binary_value",
ScalarType.createVarbinaryType(ScalarType.MAX_VARBINARY_LENGTH)),
new StructField("fixed_value", ScalarType.createVarbinaryType(fixedLength)),
new StructField("tz_value", ScalarType.createTimeStampTzType(6)),
new StructField("metric", metricType),
- new StructField("events", ArrayType.create(
- ScalarType.createVarbinaryType(16), true)),
+ new StructField("events", ArrayType.create(Type.UUID, true)),
new StructField("attrs", new MapType(
mapKeyType, ScalarType.createTimeStampTzType(6))));
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergTimestampNsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergTimestampNsTest.java
new file mode 100644
index 00000000000..ba912e89470
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergTimestampNsTest.java
@@ -0,0 +1,247 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg;
+
+import org.apache.doris.analysis.BinaryPredicate;
+import org.apache.doris.analysis.CompoundPredicate;
+import org.apache.doris.analysis.IntLiteral;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.analysis.StringLiteral;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.nereids.types.TimeStampNsType;
+
+import org.apache.iceberg.HasTableOperations;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.util.Collections;
+import java.util.Set;
+
+public class IcebergTimestampNsTest {
+ private Table createTable(Schema schema, PartitionSpec spec) {
+ InMemoryCatalog catalog = new InMemoryCatalog();
+ catalog.initialize("test", Collections.emptyMap());
+ catalog.createNamespace(Namespace.of("db"));
+ return catalog.createTable(TableIdentifier.of("db", "events"), schema, spec,
+ Collections.singletonMap("format-version", "3"));
+ }
+
+ @Test
+ public void roundTripMappingAndNestedTypes() {
+ Assertions.assertEquals(Type.TIMESTAMP_NS,
+ IcebergUtils.icebergTypeToDorisType(Types.TimestampNanoType.withoutZone(), false, false));
+ Assertions.assertEquals(Type.UNSUPPORTED,
+ IcebergUtils.icebergTypeToDorisType(Types.TimestampNanoType.withZone(), false, false));
+ Assertions.assertEquals(Types.TimestampNanoType.withoutZone(),
+ new DorisTypeToIcebergType().atomic(Type.TIMESTAMP_NS));
+ org.apache.doris.catalog.ArrayType array = (org.apache.doris.catalog.ArrayType)
+ IcebergUtils.icebergTypeToDorisType(
+ Types.ListType.ofOptional(1, Types.TimestampNanoType.withoutZone()), false, false);
+ Assertions.assertEquals(Type.TIMESTAMP_NS, array.getItemType());
+ }
+
+ @Test
+ public void partitionsDefaultsAndLiteralsPreserveSignedNanoseconds() {
+ for (long nanos : new long[] {Long.MIN_VALUE, -1, 0, 1, 123456789, Long.MAX_VALUE}) {
+ String text = IcebergUtils.serializePartitionValue(Types.TimestampNanoType.withoutZone(), nanos, "UTC");
+ Assertions.assertEquals(nanos, IcebergTimestampUtils.parseNanos(text, ZoneOffset.UTC));
+ Assertions.assertEquals(nanos, IcebergUtils.parseIcebergLiteral(
+ text, Types.TimestampNanoType.withoutZone()).value());
+ Assertions.assertEquals(nanos, IcebergUtils.parseIcebergLiteral(
+ Long.toString(nanos), Types.TimestampNanoType.withoutZone()).value());
+ org.apache.doris.nereids.trees.expressions.Expression literal = IcebergWriteSchemaContext.toDorisExpression(
+ Types.TimestampNanoType.withoutZone(), nanos, TimeStampNsType.INSTANCE, false, false);
+ Assertions.assertEquals(nanos, IcebergTimestampUtils.parseNanos(
+ ((org.apache.doris.nereids.trees.expressions.literal.Literal) literal).getStringValue(),
+ ZoneOffset.UTC));
+ Schema schema = new Schema(Types.NestedField.optional(1, "ts", Types.TimestampNanoType.withoutZone()));
+ Object converted = IcebergUtils.extractDorisLiteral(
+ Types.TimestampNanoType.withoutZone(), new StringLiteral(text));
+ org.apache.iceberg.expressions.Expression bound = Binder.bind(schema.asStruct(),
+ org.apache.iceberg.expressions.Expressions.equal("ts", converted), true);
+ Assertions.assertEquals(nanos, ((BoundPredicate<?>) bound).asLiteralPredicate().literal().value());
+ }
+ Assertions.assertThrows(ArithmeticException.class, () -> IcebergTimestampUtils.parseNanos(
+ "2262-04-11 23:47:16.854775808", ZoneOffset.UTC));
+ }
+
+ @Test
+ public void offsetAndDateOnlyPartitionsMatchSinkCasts() {
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos("2024-01-01 00:00:00", ZoneOffset.UTC),
+ IcebergTimestampUtils.parseNanos("2024-01-01", ZoneOffset.UTC));
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos(
+ "2023-12-31 16:00:00.123456789", ZoneOffset.UTC), IcebergTimestampUtils.parseNanos(
+ "2024-01-01 00:00:00.123456789+08:00", ZoneOffset.UTC));
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos(
+ "2024-01-01 00:00:00.123456789", ZoneOffset.UTC), IcebergTimestampUtils.parseNanos(
+ "2024-01-01 00:00:00.123456789+08:00", ZoneId.of("Asia/Shanghai")));
+ }
+
+ @Test
+ public void rawRewriteOffsetLiteralUsesSessionCivilTime() throws Exception {
+ Schema schema = new Schema(Types.NestedField.optional(1, "ts", Types.TimestampNanoType.withoutZone()));
+ org.apache.doris.qe.ConnectContext context = new org.apache.doris.qe.ConnectContext();
+ context.getSessionVariable().setTimeZone("Asia/Shanghai");
+ context.setThreadLocalInfo();
+ try {
+ org.apache.doris.nereids.trees.expressions.Expression predicate =
+ new org.apache.doris.nereids.trees.expressions.EqualTo(
+ new org.apache.doris.nereids.analyzer.UnboundSlot("ts"),
+ new org.apache.doris.nereids.trees.expressions.literal.StringLiteral(
+ "2024-01-01 00:00:00.123456789+08:00"));
+ org.apache.iceberg.expressions.Expression converted =
+ IcebergNereidsUtils.convertNereidsToIcebergExpression(predicate, schema);
+ BoundPredicate<?> bound = (BoundPredicate<?>) Binder.bind(schema.asStruct(), converted, true);
+ // Rewrite receives uncoerced SQL strings, unlike already-normalized metadata timestamps.
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos(
+ "2024-01-01 00:00:00.123456789", ZoneOffset.UTC),
+ bound.asLiteralPredicate().literal().value());
+ } finally {
+ org.apache.doris.qe.ConnectContext.remove();
+ }
+ }
+
+ @Test
+ public void historicalDateMetricsAndNegationCannotLoseRows() throws Exception {
+ Schema date = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "ts", Types.DateType.get()));
+ Table table = createTable(date, PartitionSpec.unpartitioned());
+ TableOperations ops = ((HasTableOperations) table).operations();
+ TableMetadata old = ops.current();
+ ops.commit(old, TableMetadata.buildFrom(old).setCurrentSchema(new Schema(
+ Types.NestedField.optional(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "ts", Types.TimestampNanoType.withoutZone())), 2).build());
+ table.refresh();
+ Set<Integer> unsafe = IcebergTimestampUtils.unsafeNanosFields(table, table.schema());
+ Assertions.assertEquals(Collections.singleton(2), unsafe);
+ BinaryPredicate ts = new BinaryPredicate(BinaryPredicate.Operator.EQ,
+ new SlotRef(null, "ts"), new StringLiteral("2024-01-02"));
+ BinaryPredicate id = new BinaryPredicate(BinaryPredicate.Operator.GT,
+ new SlotRef(null, "id"), new IntLiteral(0));
+ CompoundPredicate not = new CompoundPredicate(CompoundPredicate.Operator.NOT,
+ new CompoundPredicate(CompoundPredicate.Operator.AND, ts, id), null);
+ // Negating an unsafe conjunct must not turn NOT(ts=... AND id>0) into NOT(id>0).
+ Assertions.assertNull(IcebergUtils.convertToIcebergExpr(not, table.schema(), unsafe));
+ Assertions.assertNotNull(IcebergUtils.convertToIcebergExpr(id, table.schema(), unsafe));
+ Table cleaned = org.mockito.Mockito.spy(table);
+ org.mockito.Mockito.doReturn(Collections.singletonMap(1, table.schema())).when(cleaned).schemas();
+ Assertions.assertEquals(unsafe, IcebergTimestampUtils.unsafeNanosFields(cleaned, table.schema()));
+ }
+
+ @Test
+ public void failedNanosLiteralCannotBeNegatedAfterPartialConversion() throws Exception {
+ Schema schema = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "ts", Types.TimestampNanoType.withoutZone()));
+ BinaryPredicate invalid = new BinaryPredicate(BinaryPredicate.Operator.EQ,
+ new SlotRef(null, "ts"), new StringLiteral("2262-04-11 23:47:16.854775808"));
+ BinaryPredicate id = new BinaryPredicate(BinaryPredicate.Operator.GT,
+ new SlotRef(null, "id"), new IntLiteral(0));
+ CompoundPredicate partial = new CompoundPredicate(CompoundPredicate.Operator.AND, invalid, id);
+ Assertions.assertNotNull(IcebergUtils.convertToIcebergExpr(partial, schema));
+ Assertions.assertNull(IcebergUtils.convertToIcebergExpr(
+ new CompoundPredicate(CompoundPredicate.Operator.NOT, partial, null), schema));
+ Assertions.assertNull(IcebergUtils.extractDorisLiteral(
+ Types.TimestampNanoType.withoutZone(), new StringLiteral("invalid")));
+ Assertions.assertNull(IcebergUtils.extractDorisLiteral(
+ Types.TimestampNanoType.withoutZone(), new org.apache.doris.analysis.NullLiteral()));
+ }
+
+ @Test
+ public void staticOverwriteUsesNanosAndRejectsHistoricalDateBounds() throws Exception {
+ Schema schema = new Schema(Types.NestedField.optional(1, "ts", Types.TimestampNanoType.withoutZone()));
+ PartitionSpec spec = PartitionSpec.builderFor(schema).identity("ts").build();
+ Table table = createTable(schema, spec);
+ IcebergTransaction transaction = new IcebergTransaction(null);
+ Field field = IcebergTransaction.class.getDeclaredField("table");
+ field.setAccessible(true);
+ field.set(transaction, table);
+ org.apache.iceberg.expressions.Expression filter = transaction.buildPartitionFilter(
+ Collections.singletonMap("ts", "2024-01-02 03:04:05.123456789"), spec, table.schema());
+ Assertions.assertEquals(1704164645123456789L,
+ ((UnboundPredicate<?>) filter).literal().value());
+ TableOperations ops = ((HasTableOperations) table).operations();
+ TableMetadata old = ops.current();
+ ops.commit(old, TableMetadata.buildFrom(old).setCurrentSchema(new Schema(
+ Types.NestedField.optional(1, "ts", Types.DateType.get())), 1).build());
+ old = ops.current();
+ ops.commit(old, TableMetadata.buildFrom(old).setCurrentSchema(schema, 1).build());
+ table.refresh();
+ Assertions.assertThrows(IllegalArgumentException.class, () -> transaction.buildPartitionFilter(
+ Collections.singletonMap("ts", "2024-01-02"), spec, schema));
+ }
+
+ @Test
+ public void calendarPartitionsAvoidOutOfRangeMvBoundaries() {
+ Schema schema = new Schema(Types.NestedField.optional(1, "ts", Types.TimestampNanoType.withoutZone()));
+ for (PartitionSpec spec : new PartitionSpec[] {PartitionSpec.builderFor(schema).year("ts").build(),
+ PartitionSpec.builderFor(schema).month("ts").build(),
+ PartitionSpec.builderFor(schema).day("ts").build(),
+ PartitionSpec.builderFor(schema).hour("ts").build()}) {
+ Assertions.assertFalse(IcebergUtils.isValidRelatedTable(createTable(schema, spec)));
+ }
+ }
+
+ @Test
+ public void lossyTimestampCastsRemainResidual() throws Exception {
+ Schema schema = new Schema(Types.NestedField.optional(1, "ts", Types.TimestampNanoType.withoutZone()));
+ SlotRef slot = new SlotRef(null, "ts");
+ slot.setType(Type.TIMESTAMP_NS);
+ org.apache.doris.analysis.CastExpr cast = new org.apache.doris.analysis.CastExpr(
+ org.apache.doris.catalog.ScalarType.createDatetimeV2Type(6), slot, null);
+ StringLiteral value = new StringLiteral("2024-01-02 03:04:05.123457");
+ // The rounded equality can match nanos that an exact Iceberg predicate would prune.
+ Assertions.assertNull(IcebergUtils.convertToIcebergExpr(
+ new BinaryPredicate(BinaryPredicate.Operator.EQ, cast, value), schema));
+ Assertions.assertNull(IcebergUtils.convertToIcebergExpr(
+ new BinaryPredicate(BinaryPredicate.Operator.EQ, value, cast), schema));
+ for (boolean notIn : new boolean[] {false, true}) {
+ Assertions.assertNull(IcebergUtils.convertToIcebergExpr(new org.apache.doris.analysis.InPredicate(
+ cast, Collections.singletonList(value), notIn), schema));
+ }
+ Assertions.assertNotNull(IcebergUtils.convertToIcebergExpr(
+ new BinaryPredicate(BinaryPredicate.Operator.EQ, slot, value), schema));
+ Assertions.assertNotNull(IcebergUtils.convertToIcebergExpr(new BinaryPredicate(BinaryPredicate.Operator.EQ,
+ new org.apache.doris.analysis.CastExpr(Type.TIMESTAMP_NS, slot, null), value), schema));
+ }
+
+ @Test
+ public void compactStaticPartitionLiteralsUseSinkParsing() {
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos("2024-01-02", ZoneOffset.UTC),
+ IcebergTimestampUtils.parseNanos("20240102", ZoneOffset.UTC));
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos("2024-01-02 03:04:05", ZoneOffset.UTC),
+ IcebergTimestampUtils.parseNanos("20240102030405", ZoneOffset.UTC));
+ Assertions.assertEquals(IcebergTimestampUtils.parseNanos("2024-01-02 03:04:05.123456790", ZoneOffset.UTC),
+ IcebergTimestampUtils.parseNanos("2024-01-02 03:04:05.1234567895", ZoneOffset.UTC));
+ }
+
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
index 145fead12ee..bdb5e593f7c 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.datasource.iceberg;
import org.apache.doris.analysis.DateLiteral;
import org.apache.doris.analysis.TableScanParams;
import org.apache.doris.analysis.TableSnapshot;
+import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.PartitionKey;
@@ -348,9 +349,12 @@ public class IcebergUtilsTest {
Assert.assertTrue(binary.isVarbinaryType());
Assert.assertEquals(ScalarType.MAX_VARBINARY_LENGTH, ((ScalarType) binary).getLength());
- Type uuid = IcebergUtils.icebergTypeToDorisType(Types.UUIDType.get(), false, false);
- Assert.assertTrue(uuid.isVarbinaryType());
- Assert.assertEquals(16, ((ScalarType) uuid).getLength());
+ for (boolean mapping : Arrays.asList(false, true)) {
+ Assert.assertEquals(Type.UUID,
+ IcebergUtils.icebergTypeToDorisType(Types.UUIDType.get(), mapping, false));
+ Assert.assertEquals(ArrayType.create(Type.UUID, true), IcebergUtils.icebergTypeToDorisType(
+ Types.ListType.ofOptional(1, Types.UUIDType.get()), mapping, false));
+ }
}
@Test
@@ -942,12 +946,12 @@ public class IcebergUtilsTest {
Assert.assertEquals("7", serializedDefaults.get(1));
Assert.assertEquals("2024-01-01 00:00:00.123456", serializedDefaults.get(2));
Assert.assertEquals("2024-01-01 00:00:00.123456+00:00", serializedDefaults.get(6));
- Assert.assertEquals("AAAAAAAAAAAAAAAAAAAAAA==", serializedDefaults.get(3));
+ Assert.assertEquals("00000000-0000-0000-0000-000000000000", serializedDefaults.get(3));
Assert.assertEquals("AAEC/w==", serializedDefaults.get(4));
Assert.assertEquals("AwIBAA==", serializedDefaults.get(5));
Map<Integer, String> base64Defaults = IcebergUtils.getBase64EncodedInitialDefaults(schema);
- Assert.assertEquals("AAAAAAAAAAAAAAAAAAAAAA==", base64Defaults.get(3));
+ Assert.assertFalse(base64Defaults.containsKey(3));
Assert.assertEquals("AAEC/w==", base64Defaults.get(4));
Assert.assertEquals("AwIBAA==", base64Defaults.get(5));
}
@@ -1006,7 +1010,7 @@ public class IcebergUtilsTest {
Assert.assertFalse(defaults.containsKey(4));
Assert.assertEquals(Collections.singleton(3),
IcebergUtils.getBase64EncodedInitialDefaults(schema).keySet());
- Assert.assertEquals(ImmutableSet.of(3, 4), IcebergUtils.getBinaryLikeFieldIds(schema));
+ Assert.assertEquals(ImmutableSet.of(3), IcebergUtils.getBinaryLikeFieldIds(schema));
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUuidMappingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUuidMappingTest.java
new file mode 100644
index 00000000000..753b5e81869
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUuidMappingTest.java
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg;
+
+import org.apache.doris.catalog.Type;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.UuidLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.MapType;
+import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+import org.apache.doris.nereids.types.UuidType;
+
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.UUID;
+
+public class IcebergUuidMappingTest {
+ @Test
+ public void uuidDefaultsAndWritesRetainLogicalType() {
+ UUID value = UUID.fromString("00112233-4455-6677-8899-aabbccddeeff");
+ Types.NestedField field = Types.NestedField.optional("u").withId(1)
+ .ofType(Types.UUIDType.get()).withInitialDefault(value).build();
+ Assert.assertEquals(value.toString(), IcebergUtils.getSerializedInitialDefault(field, false));
+ Assert.assertFalse(IcebergUtils.getBase64EncodedInitialDefaults(new Schema(field)).containsKey(1));
+ Assert.assertEquals(Types.UUIDType.get(), new DorisTypeToIcebergType().atomic(Type.UUID));
+ Assert.assertEquals(value, IcebergUtils.parsePartitionValueFromString(
+ value.toString().replace("-", ""), Types.UUIDType.get()));
+ Assert.assertEquals(value, IcebergUtils.parsePartitionValueFromString(
+ value.toString().toUpperCase(java.util.Locale.ROOT), Types.UUIDType.get()));
+ Assert.assertEquals(new UuidLiteral(value.toString()), IcebergWriteSchemaContext.toDorisExpression(
+ Types.UUIDType.get(), value, UuidType.INSTANCE, false, false));
+ }
+
+ @Test
+ public void uuidWriteCastsRejectInvalidTextAtEveryDepth() {
+ SlotReference text = new SlotReference("value", StringType.INSTANCE);
+ Assert.assertTrue(((Cast) IcebergUuidWriteConverter.convert(text, UuidType.INSTANCE)).isStrict());
+ Assert.assertTrue(IcebergUuidWriteConverter.convert(
+ new SlotReference("items", ArrayType.of(StringType.INSTANCE)),
+ ArrayType.of(UuidType.INSTANCE)).anyMatch(e -> e instanceof Cast && ((Cast) e).isStrict()));
+ Assert.assertTrue(IcebergUuidWriteConverter.convert(
+ new SlotReference("map", MapType.of(StringType.INSTANCE, StringType.INSTANCE)),
+ MapType.of(UuidType.INSTANCE, UuidType.INSTANCE)).anyMatch(e -> e instanceof Cast && ((Cast) e).isStrict()));
+ Assert.assertTrue(IcebergUuidWriteConverter.convert(
+ new SlotReference("record", new StructType(Collections.singletonList(
+ new StructField("u", StringType.INSTANCE, true, "")))),
+ new StructType(Collections.singletonList(
+ new StructField("u", UuidType.INSTANCE, true, "")))).anyMatch(
+ e -> e instanceof Cast && ((Cast) e).isStrict()));
+ Assert.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class,
+ () -> IcebergUuidWriteConverter.convert(
+ new org.apache.doris.nereids.trees.expressions.literal.StringLiteral("invalid-uuid"),
+ UuidType.INSTANCE));
+ Expression uuid = new SlotReference("uuid", UuidType.INSTANCE);
+ Assert.assertSame(uuid, IcebergUuidWriteConverter.convert(uuid, UuidType.INSTANCE));
+ Assert.assertEquals(new NullLiteral(UuidType.INSTANCE),
+ IcebergUuidWriteConverter.convert(NullLiteral.INSTANCE, UuidType.INSTANCE));
+ Assert.assertSame(text, IcebergUuidWriteConverter.convert(text, StringType.INSTANCE));
+ }
+
+ @Test
+ public void mapUuidValidationPreservesOrdinarySiblingCasts() {
+ SlotReference source = new SlotReference("m", MapType.of(StringType.INSTANCE, BigIntType.INSTANCE));
+ MapType target = MapType.of(UuidType.INSTANCE, IntegerType.INSTANCE);
+ Expression result = IcebergUuidWriteConverter.convert(source, target);
+ Assert.assertEquals(target, result.getDataType());
+ Assert.assertTrue(result instanceof Cast);
+ Assert.assertFalse(((Cast) result).isStrict());
+ Cast validation = (Cast) result.child(0);
+ Assert.assertTrue(validation.isStrict());
+ Assert.assertEquals(MapType.of(UuidType.INSTANCE, BigIntType.INSTANCE), validation.getDataType());
+ Assert.assertSame(source, validation.child(0));
+ }
+
+ @Test
+ public void mapValidationPreservesNestedSiblingTypesAndNullability() {
+ StructType sourceValue = new StructType(Arrays.asList(
+ new StructField("u", StringType.INSTANCE, true, ""),
+ new StructField("n", BigIntType.INSTANCE, true, "")));
+ StructType targetValue = new StructType(Arrays.asList(
+ new StructField("u", UuidType.INSTANCE, true, ""),
+ new StructField("n", IntegerType.INSTANCE, true, "")));
+ SlotReference source = new SlotReference("m", MapType.of(StringType.INSTANCE, ArrayType.of(sourceValue)));
+ Expression result = IcebergUuidWriteConverter.convert(source,
+ MapType.of(StringType.INSTANCE, ArrayType.of(targetValue)));
+ Assert.assertFalse(((Cast) result).isStrict());
+ Cast validation = (Cast) result.child(0);
+ Assert.assertTrue(validation.isStrict());
+ Assert.assertSame(source, validation.child(0));
+ Assert.assertEquals(MapType.of(StringType.INSTANCE, ArrayType.of(new StructType(Arrays.asList(
+ new StructField("u", UuidType.INSTANCE, true, ""),
+ new StructField("n", BigIntType.INSTANCE, true, ""))))), validation.getDataType());
+ }
+
+ @Test
+ public void serializedContainersValidateUuidTextBeforeOrdinaryCoercion() {
+ SlotReference source = new SlotReference("text", StringType.INSTANCE);
+ for (DataType target : Arrays.asList(ArrayType.of(UuidType.INSTANCE),
+ MapType.of(UuidType.INSTANCE, IntegerType.INSTANCE),
+ new StructType(Arrays.asList(new StructField("u", UuidType.INSTANCE, true, ""),
+ new StructField("n", IntegerType.INSTANCE, true, ""))),
+ ArrayType.of(MapType.of(StringType.INSTANCE, UuidType.INSTANCE)))) {
+ Expression result = IcebergUuidWriteConverter.convert(source, target);
+ Assert.assertEquals(target, result.getDataType());
+ Assert.assertTrue("Missing UUID validation for " + target, result.anyMatch(
+ e -> e instanceof Cast && ((Cast) e).isStrict()
+ && IcebergUuidWriteConverter.containsUuid(((Cast) e).getDataType())));
+ Assert.assertFalse("Serialized UUID text must survive initial parsing", result.anyMatch(
+ e -> e instanceof Cast && e.child(0).equals(source)
+ && IcebergUuidWriteConverter.containsUuid(((Cast) e).getDataType())));
+ }
+ }
+
+ @Test
+ public void incompatibleUuidBoundsDoNotPruneFiles() throws Exception {
+ Assert.assertNull(IcebergUtils.extractDorisLiteral(Types.UUIDType.get(),
+ new org.apache.doris.analysis.UuidLiteral("80000000-0000-0000-0000-000000000000")));
+ Assert.assertNull(IcebergUtils.extractDorisLiteral(Types.UUIDType.get(),
+ new org.apache.doris.analysis.StringLiteral("80000000-0000-0000-0000-000000000000")));
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContextTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContextTest.java
index f587a4bc411..22c30673ee2 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContextTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergWriteSchemaContextTest.java
@@ -177,15 +177,10 @@ public class IcebergWriteSchemaContextTest {
stringValue(context.resolveWriteDefault(columns.get("timestamp_col"))));
Assertions.assertEquals("2025-01-18 01:02:03.654321+00:00",
stringValue(context.resolveWriteDefault(columns.get("timestamptz_col"))));
- Assertions.assertArrayEquals(
- ByteBuffer.allocate(16)
- .putLong(UUID.fromString("123e4567-e89b-12d3-a456-426614174001")
- .getMostSignificantBits())
- .putLong(UUID.fromString("123e4567-e89b-12d3-a456-426614174001")
- .getLeastSignificantBits())
- .array(),
- (byte[]) ((VarBinaryLiteral) context.resolveWriteDefault(
- columns.get("uuid_col"))).getValue());
+ Assertions.assertEquals(
+ new org.apache.doris.nereids.trees.expressions.literal.UuidLiteral(
+ "123e4567-e89b-12d3-a456-426614174001"),
+ context.resolveWriteDefault(columns.get("uuid_col")));
Assertions.assertArrayEquals(new byte[] {0x0a, 0x0b, 0x0c, 0x0d},
(byte[]) ((VarBinaryLiteral) context.resolveWriteDefault(
columns.get("fixed_col"))).getValue());
@@ -248,10 +243,8 @@ public class IcebergWriteSchemaContextTest {
Expression uuid = IcebergWriteSchemaContext.toDorisExpression(
Types.UUIDType.get(), UUID.fromString("123e4567-e89b-12d3-a456-426614174000"),
uuidTarget, false, false);
- UUID uuidValue = UUID.fromString("123e4567-e89b-12d3-a456-426614174000");
- byte[] uuidBytes = ByteBuffer.allocate(16).putLong(uuidValue.getMostSignificantBits())
- .putLong(uuidValue.getLeastSignificantBits()).array();
- Assertions.assertArrayEquals(uuidBytes, (byte[]) ((VarBinaryLiteral) uuid).getValue());
+ Assertions.assertEquals(new org.apache.doris.nereids.trees.expressions.literal.UuidLiteral(
+ "123e4567-e89b-12d3-a456-426614174000"), uuid);
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlannerTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlannerTest.java
index 230a7223d63..9c00b66b090 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlannerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlannerTest.java
@@ -99,6 +99,32 @@ public class RewriteDataFilePlannerTest {
Mockito.when(mockTableScan.ignoreResiduals()).thenReturn(mockTableScan);
}
+ @Test
+ public void testQuotedPromotedTimestampRejectsUnsafeRewritePruning() throws Exception {
+ Mockito.when(mockTable.newScan()).thenReturn(mockTableScan);
+ Mockito.when(mockTableScan.filter(Mockito.any())).thenReturn(mockTableScan);
+ Mockito.when(mockTableScan.planFiles()).thenReturn(
+ CloseableIterable.withNoopClose(Collections.emptyList()));
+ for (String name : Arrays.asList("ts.day", "has space", "tick`name")) {
+ Schema date = new Schema(0, Types.NestedField.optional(1, name, Types.DateType.get()));
+ Schema nanos = new Schema(1,
+ Types.NestedField.optional(1, name, Types.TimestampNanoType.withoutZone()));
+ Mockito.when(mockTable.schema()).thenReturn(nanos);
+ Mockito.when(mockTable.schemas()).thenReturn(
+ com.google.common.collect.ImmutableMap.of(0, date, 1, nanos));
+ org.apache.doris.nereids.trees.expressions.Expression predicate =
+ new org.apache.doris.nereids.parser.NereidsParser().parseExpression(
+ "`" + name.replace("`", "``") + "` = '2024-01-02'");
+ RewriteDataFilePlanner.Parameters parameters = new RewriteDataFilePlanner.Parameters(
+ 128L, 64L, 256L, 2, false, 512L, 3, 0.1, 0L, Optional.of(predicate));
+ UserException error = Assertions.assertThrows(UserException.class,
+ () -> new RewriteDataFilePlanner(parameters).planAndOrganizeTasks(mockTable));
+ Assertions.assertTrue(error.getMessage().contains("cannot prune historical DATE bounds"),
+ error.getMessage());
+ }
+ Mockito.verify(mockTableScan, Mockito.never()).filter(Mockito.any());
+ }
+
@Test
public void testParametersGetters() {
Assertions.assertEquals(128 * 1024 * 1024L, defaultParameters.getTargetFileSizeBytes());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/source/IcebergScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/source/IcebergScanNodeTest.java
index 84d5f625d53..f997dce4a91 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/source/IcebergScanNodeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/source/IcebergScanNodeTest.java
@@ -2454,11 +2454,45 @@ public class IcebergScanNodeTest {
Assert.assertEquals(2, node.getScanColumns(new Schema(fields)).size());
}
+ @Test
+ public void testSchemaCarrierRetainsReferencedHistoricalNanosField() throws Exception {
+ Types.NestedField id = Types.NestedField.required(1, "id", Types.LongType.get());
+ Types.NestedField nanos = Types.NestedField.optional(
+ 9, "equality_nanos", Types.TimestampNanoType.withoutZone());
+ Schema historicalSchema = new Schema(1, ImmutableList.of(id, nanos));
+ Schema currentSchema = new Schema(2, ImmutableList.of(id));
+ Snapshot historicalSnapshot = mockSnapshot(1000L, historicalSchema, null);
+ Snapshot currentSnapshot = mockSnapshot(1001L, currentSchema, 1000L);
+ TableMetadata metadata = Mockito.mock(TableMetadata.class);
+ Mockito.when(metadata.schemas()).thenReturn(ImmutableList.of(historicalSchema, currentSchema));
+ Mockito.when(metadata.schemasById()).thenReturn(ImmutableMap.of(
+ historicalSchema.schemaId(), historicalSchema,
+ currentSchema.schemaId(), currentSchema));
+ Mockito.when(metadata.snapshot(1000L)).thenReturn(historicalSnapshot);
+ TableOperations operations = Mockito.mock(TableOperations.class);
+ Mockito.when(operations.current()).thenReturn(metadata);
+ BaseTable table = new BaseTable(operations, "test");
+ TableScan tableScan = Mockito.mock(TableScan.class);
+ Mockito.when(tableScan.snapshot()).thenReturn(currentSnapshot);
+
+ TestIcebergScanNode node = new TestIcebergScanNode(new SessionVariable());
+ setIcebergTable(node, table);
+ node.setTableScan(tableScan);
+
+ List<Types.NestedField> fields = node.getSchemaFieldsForScan(
+ currentSchema, ImmutableSet.of(9));
+ // Dropped equality fields remain readable when their nanos type is supported.
+ List<Column> columns = node.getScanColumns(new Schema(fields));
+ Assert.assertEquals(2, columns.size());
+ Assert.assertEquals(Type.TIMESTAMP_NS, columns.get(1).getType());
+ }
+
@Test(expected = IllegalArgumentException.class)
public void testSchemaCarrierRejectsReferencedUnsupportedHistoricalField() throws Exception {
Types.NestedField id = Types.NestedField.required(1, "id", Types.LongType.get());
+ // UNKNOWN still exercises rejection now that unzoned nanos have a valid Doris mapping.
Types.NestedField unsupported = Types.NestedField.optional(
- 9, "equality_nanos", Types.TimestampNanoType.withoutZone());
+ 9, "unknown_equality_field", Types.UnknownType.get());
Schema historicalSchema = new Schema(1, ImmutableList.of(id, unsupported));
Schema currentSchema = new Schema(2, ImmutableList.of(id));
Snapshot historicalSnapshot = mockSnapshot(1000L, historicalSchema, null);
@@ -3744,6 +3778,148 @@ public class IcebergScanNodeTest {
}
}
+ @Test
+ public void testManifestCacheKeepsEqualityDeleteAfterDateToNanosPromotion() throws Exception {
+ checkDateToNanosEqualityDeletes(false, true);
+ }
+
+ @Test
+ public void testManifestCacheKeepsDroppedDateToNanosEqualityDelete() throws Exception {
+ checkDateToNanosEqualityDeletes(true, true);
+ }
+
+ @Test
+ public void testSdkKeepsDateToNanosEqualityDelete() throws Exception {
+ checkDateToNanosEqualityDeletes(false, false);
+ }
+
+ @Test
+ public void testSdkKeepsDroppedDateToNanosEqualityDelete() throws Exception {
+ checkDateToNanosEqualityDeletes(true, false);
+ }
+
+ private void checkDateToNanosEqualityDeletes(boolean dropped, boolean manifestCache) throws Exception {
+ checkDateToNanosEqualityDeletes(dropped, manifestCache, false);
+ }
+
+ @Test
+ public void testHistoricalDateDeleteBoundsAfterDroppingPromotedKey() throws Exception {
+ checkDateToNanosEqualityDeletes(true, false, true);
+ checkDateToNanosEqualityDeletes(true, true, true);
+ }
+
+ private void checkDateToNanosEqualityDeletes(boolean dropped, boolean manifestCache,
+ boolean dateDelete) throws Exception {
+ Schema schema = new Schema(Types.NestedField.required(1, "ts", Types.DateType.get()),
+ Types.NestedField.optional(2, "id", Types.IntegerType.get()));
+ String tableLocation = temporaryFolder.getRoot().toPath()
+ .resolve("manifest_cache_date_to_nanos_" + manifestCache + "_" + dateDelete).toUri().toString();
+ Table table = new HadoopTables(new Configuration()).create(
+ schema, PartitionSpec.unpartitioned(), SortOrder.unsorted(),
+ ImmutableMap.of(TableProperties.FORMAT_VERSION, "3"), tableLocation);
+ Map<Integer, java.nio.ByteBuffer> dateBounds = ImmutableMap.of(1,
+ org.apache.iceberg.types.Conversions.toByteBuffer(Types.DateType.get(), 19724));
+ Map<Integer, java.nio.ByteBuffer> nanosBounds = ImmutableMap.of(1,
+ org.apache.iceberg.types.Conversions.toByteBuffer(
+ Types.TimestampNanoType.withoutZone(), 1704153600000000000L));
+ DataFile dataFile = DataFiles.builder(table.spec())
+ .withPath(tableLocation + "/data/old.parquet")
+ .withFormat(FileFormat.PARQUET)
+ .withFileSizeInBytes(10)
+ .withMetrics(new org.apache.iceberg.Metrics(1L, null, ImmutableMap.of(1, 1L),
+ ImmutableMap.of(1, 0L), null,
+ dateDelete ? nanosBounds : dateBounds, dateDelete ? nanosBounds : dateBounds))
+ .build();
+ if (!dateDelete) {
+ table.newFastAppend().appendFile(dataFile).commit();
+ }
+ TableOperations operations = ((org.apache.iceberg.HasTableOperations) table).operations();
+ TableMetadata before = operations.current();
+ operations.commit(before, TableMetadata.buildFrom(before).setCurrentSchema(new Schema(
+ Types.NestedField.required(1, "ts", Types.TimestampNanoType.withoutZone()),
+ Types.NestedField.optional(2, "id", Types.IntegerType.get())), 2).build());
+ table.refresh();
+ if (dateDelete) {
+ table.newFastAppend().appendFile(dataFile).commit();
+ }
+ table.newRowDelta().addDeletes(FileMetadata.deleteFileBuilder(table.spec())
+ .ofEqualityDeletes(1)
+ .withPath(tableLocation + "/data/eq-delete.parquet")
+ .withFormat(FileFormat.PARQUET)
+ .withFileSizeInBytes(10)
+ .withMetrics(new org.apache.iceberg.Metrics(1L, null, ImmutableMap.of(1, 1L),
+ ImmutableMap.of(1, 0L), null,
+ dateDelete ? dateBounds : nanosBounds, dateDelete ? dateBounds : nanosBounds))
+ .build()).commit();
+
+ if (dropped) {
+ table.updateSchema().deleteColumn("ts").commit();
+ }
+
+ TestIcebergScanNode node = new TestIcebergScanNode(new SessionVariable());
+ IcebergSource source = Mockito.mock(IcebergSource.class);
+ Mockito.when(source.getCatalog()).thenReturn(Mockito.mock(ExternalCatalog.class));
+ Mockito.when(source.getTargetTable()).thenReturn(Mockito.mock(ExternalTable.class));
+ setPrivateField(node, "source", source);
+ setPrivateField(node, "icebergTable", table);
+ Env env = Mockito.mock(Env.class);
+ Mockito.when(env.getExtMetaCacheMgr()).thenReturn(Mockito.mock(ExternalMetaCacheMgr.class));
+ try (MockedStatic<Env> mockedEnv = Mockito.mockStatic(Env.class);
+ MockedStatic<IcebergManifestCacheLoader> loader =
+ Mockito.mockStatic(IcebergManifestCacheLoader.class)) {
+ mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
+ loader.when(() -> IcebergManifestCacheLoader.loadDeleteFilesWithCache(Mockito.any(), Mockito.any(),
+ Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any()))
+ .thenAnswer(invocation -> {
+ List<DeleteFile> files = new ArrayList<>();
+ try (CloseableIterable<DeleteFile> reader = ManifestFiles.readDeleteManifest(
+ invocation.getArgument(2), table.io(), table.specs())) {
+ reader.forEach(file -> files.add(file.copy()));
+ }
+ return ManifestCacheValue.forDeleteFiles(files);
+ });
+ loader.when(() -> IcebergManifestCacheLoader.loadDataFilesWithCache(Mockito.any(), Mockito.any(),
+ Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any()))
+ .thenAnswer(invocation -> {
+ List<DataFile> files = new ArrayList<>();
+ try (CloseableIterable<DataFile> reader = ManifestFiles.read(
+ invocation.getArgument(2), table.io(), table.specs())) {
+ reader.forEach(file -> files.add(file.copy()));
+ }
+ return ManifestCacheValue.forDataFiles(files);
+ });
+
+ TableScan scan = org.apache.iceberg.DorisDataTableScan.wrap(table.newScan().project(table.schema()));
+ Assert.assertEquals(dropped, scan.schema().findField("ts") == null);
+ List<FileScanTask> tasks;
+ if (manifestCache) {
+ tasks = node.loadFileScanTasksWithManifestCache(scan, table.currentSnapshot());
+ } else {
+ tasks = new ArrayList<>();
+ try (CloseableIterable<FileScanTask> planned = scan.planFiles()) {
+ planned.forEach(tasks::add);
+ }
+ }
+
+ // Historical DATE bounds are days, not nanos; they must not exclude the later delete.
+ Assert.assertEquals(1, tasks.size());
+ Assert.assertEquals(1, tasks.get(0).deletes().size());
+ if (!dateDelete) {
+ Map<Integer, java.nio.ByteBuffer> otherNanos = ImmutableMap.of(1,
+ org.apache.iceberg.types.Conversions.toByteBuffer(
+ Types.TimestampNanoType.withoutZone(), 1704240000000000000L));
+ DataFile disjoint = DataFiles.builder(table.spec()).withPath("disjoint.parquet")
+ .withFormat(FileFormat.PARQUET).withFileSizeInBytes(10)
+ .withMetrics(new org.apache.iceberg.Metrics(1L, null, ImmutableMap.of(1, 1L),
+ ImmutableMap.of(1, 0L), null, otherNanos, otherNanos)).build();
+ // Compatible nanos bounds still prune; only historical day bounds are unknown.
+ Assert.assertEquals(0, org.apache.iceberg.DeleteFileIndex
+ .builderFor(tasks.get(0).deletes()).schemasById(table.schemas())
+ .specsById(table.specs()).build().forDataFile(0L, disjoint).length);
+ }
+ }
+ }
+
@Test
public void testPinnedBranchUsesFrozenSnapshotWithCurrentSchema() throws Exception {
Schema snapshotSchema = new Schema(11, ImmutableList.of(
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcUuidMappingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcUuidMappingTest.java
new file mode 100644
index 00000000000..088e2a19315
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcUuidMappingTest.java
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.jdbc.client;
+
+import org.apache.doris.catalog.ArrayType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.sql.ResultSet;
+import java.sql.Types;
+
+public class JdbcUuidMappingTest {
+ private JdbcFieldSchema field(String type, int jdbcType, int dimensions) throws Exception {
+ ResultSet metadata = Mockito.mock(ResultSet.class);
+ Mockito.when(metadata.getString("TYPE_NAME")).thenReturn(type);
+ Mockito.when(metadata.getInt("DATA_TYPE")).thenReturn(jdbcType);
+ return new JdbcFieldSchema(metadata, dimensions);
+ }
+
+ @Test
+ public void nativeUuidAndNestedArrays() throws Exception {
+ JdbcPostgreSQLClient postgres = Mockito.mock(JdbcPostgreSQLClient.class, Mockito.CALLS_REAL_METHODS);
+ JdbcSQLServerClient sqlServer = Mockito.mock(JdbcSQLServerClient.class, Mockito.CALLS_REAL_METHODS);
+ JdbcTrinoClient trino = Mockito.mock(JdbcTrinoClient.class, Mockito.CALLS_REAL_METHODS);
+ Assert.assertEquals(Type.UUID, postgres.jdbcTypeToDoris(field("uuid", Types.OTHER, 0)));
+ Assert.assertEquals(Type.UUID, sqlServer.jdbcTypeToDoris(field("uniqueidentifier", Types.CHAR, 0)));
+ Assert.assertEquals(Type.UUID, trino.jdbcTypeToDoris(field("uuid", Types.OTHER, 0)));
+ Type nested = ArrayType.create(ArrayType.create(Type.UUID, true), true);
+ Assert.assertEquals(nested, postgres.jdbcTypeToDoris(field("_uuid", Types.ARRAY, 2)));
+ Assert.assertEquals(nested, trino.jdbcTypeToDoris(field("array(array(uuid))", Types.ARRAY, 2)));
+ }
+
+ @Test
+ public void logicalUuidIsPreservedInHudiAndTrinoConnector() throws Exception {
+ org.apache.avro.Schema uuid = org.apache.avro.LogicalTypes.uuid()
+ .addToSchema(org.apache.avro.Schema.create(org.apache.avro.Schema.Type.STRING));
+ Assert.assertEquals(Type.UUID,
+ org.apache.doris.datasource.hudi.HudiUtils.fromAvroHudiTypeToDorisType(uuid));
+ Assert.assertEquals(ArrayType.create(Type.UUID, true),
+ org.apache.doris.datasource.hudi.HudiUtils.fromAvroHudiTypeToDorisType(
+ org.apache.avro.Schema.createArray(uuid)));
+ Class<?> tableClass = org.apache.doris.datasource.trinoconnector.TrinoConnectorExternalTable.class;
+ java.lang.reflect.Method convert = tableClass.getDeclaredMethod(
+ "trinoConnectorTypeToDorisType", io.trino.spi.type.Type.class);
+ convert.setAccessible(true);
+ Object table = Mockito.mock(tableClass, Mockito.CALLS_REAL_METHODS);
+ Assert.assertEquals(Type.UUID, convert.invoke(table, io.trino.spi.type.UuidType.UUID));
+ Assert.assertEquals(ArrayType.create(Type.UUID, true),
+ convert.invoke(table, new io.trino.spi.type.ArrayType(io.trino.spi.type.UuidType.UUID)));
+ }
+
+ @Test
+ public void uuidInsertParametersHaveRemoteType() {
+ for (String dialect : new String[] {"trino", "presto", "postgresql", "sqlserver"}) {
+ org.apache.doris.catalog.JdbcTable table = new org.apache.doris.catalog.JdbcTable(1, "events",
+ java.util.Arrays.asList(new org.apache.doris.catalog.Column("id", Type.INT),
+ new org.apache.doris.catalog.Column("u", Type.UUID)),
+ org.apache.doris.catalog.Table.TableType.JDBC);
+ table.setJdbcTypeName(dialect);
+ table.setExternalTableName("events");
+ String sql = table.getInsertSql(java.util.Arrays.asList("id", "u"));
+ Assert.assertTrue(sql, sql.endsWith(dialect.equals("trino") || dialect.equals("presto")
+ ? "VALUES (?, CAST(? AS UUID))" : "VALUES (?, ?)"));
+ }
+ }
+
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/source/JdbcScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/source/JdbcScanNodeTest.java
index 4dd1e2ed4f6..97ed3a127ed 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/source/JdbcScanNodeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/source/JdbcScanNodeTest.java
@@ -136,6 +136,24 @@ public class JdbcScanNodeTest {
new InPredicate(slot, Arrays.asList(new VarBinaryLiteral(new byte[] {0, (byte) 0xff})), false)));
}
+ @Test
+ public void testUuidOrderingStaysLocal() throws Exception {
+ Method method = JdbcScanNode.class.getDeclaredMethod("shouldPushDownConjunct",
+ TOdbcTableType.class, Expr.class);
+ method.setAccessible(true);
+ SlotRef slot = new SlotRef(null, "u");
+ slot.setType(Type.UUID);
+ Expr literal = new org.apache.doris.analysis.UuidLiteral("80000000-0000-0000-0000-000000000000");
+ for (TOdbcTableType dialect : Arrays.asList(TOdbcTableType.SQLSERVER, TOdbcTableType.CLICKHOUSE)) {
+ for (Operator operator : Arrays.asList(Operator.LT, Operator.LE, Operator.GT, Operator.GE)) {
+ Assert.assertEquals(false, method.invoke(null, dialect,
+ new BinaryPredicate(operator, slot, literal)));
+ }
+ Assert.assertEquals(true, method.invoke(null, dialect, new BinaryPredicate(Operator.EQ, slot, literal)));
+ Assert.assertEquals(true, method.invoke(null, dialect, new BinaryPredicate(Operator.NE, slot, literal)));
+ }
+ }
+
@Mocked
private JdbcTable mockTable;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
index d55fab81086..a6cfb955c89 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.datasource.paimon;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.Type;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.UserException;
import org.apache.doris.common.security.authentication.ExecutionAuthenticator;
@@ -526,19 +527,20 @@ public class PaimonMetadataOpsTest {
}
@Test
- public void testModifyColumnPreservesRemoteTypeBehindLossyProjection()
- throws Exception {
- TimestampType remoteType = new TimestampType(9);
- DataField remoteField = new DataField(0, "ts", remoteType);
- Column projectedColumn = new Column(
- "ts", ScalarType.createDatetimeV2Type(6), true);
-
- org.apache.paimon.types.DataType requestedType =
- ops.requestedColumnType(projectedColumn, remoteField);
-
- Assert.assertTrue(requestedType instanceof TimestampType);
- Assert.assertEquals(9, ((TimestampType) requestedType).getPrecision());
- Assert.assertEquals(projectedColumn.isAllowNull(), requestedType.isNullable());
+ public void testModifyColumnPreservesRemoteTimestampPrecision() throws Exception {
+ for (int precision : new int[] {7, 8, 9}) {
+ DataField remoteField = new DataField(0, "ts", new TimestampType(precision));
+ Column projectedColumn = new Column("ts", Type.TIMESTAMP_NS, true);
+ // TIMESTAMP_NS represents all three precisions; an attribute-only ALTER must preserve them.
+ org.apache.paimon.types.DataType requestedType = ops.requestedColumnType(projectedColumn, remoteField);
+ Assert.assertTrue(requestedType instanceof TimestampType);
+ Assert.assertEquals(precision, ((TimestampType) requestedType).getPrecision());
+ Assert.assertEquals(projectedColumn.isAllowNull(), requestedType.isNullable());
+
+ // DATETIMEV2(6) is now an explicit type change, not the unchanged external projection.
+ Column reducedColumn = new Column("ts", ScalarType.createDatetimeV2Type(6), true);
+ Assert.assertEquals(6, ((TimestampType) ops.requestedColumnType(reducedColumn, remoteField)).getPrecision());
+ }
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonTimestampNsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonTimestampNsTest.java
new file mode 100644
index 00000000000..a0197696293
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonTimestampNsTest.java
@@ -0,0 +1,76 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import org.apache.doris.catalog.Type;
+
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.IntType;
+import org.apache.paimon.types.LocalZonedTimestampType;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.types.TimestampType;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class PaimonTimestampNsTest {
+ @Test
+ public void nanosecondTimestampRoundTripsWithoutTimezone() {
+ for (boolean flag : new boolean[] {false, true}) {
+ for (int precision : new int[] {0, 3, 6, 7, 8, 9}) {
+ Type mapped = PaimonUtil.paimonTypeToDorisType(new TimestampType(precision), flag, flag);
+ Assertions.assertEquals(precision > 6, mapped.isTimeStampNs());
+ Assertions.assertTrue(PaimonUtil.paimonTypeToDorisType(
+ new LocalZonedTimestampType(precision), flag, flag).isTimeStampTz());
+ }
+ }
+ Assertions.assertEquals(new TimestampType(9), new DorisToPaimonTypeVisitor().atomic(Type.TIMESTAMP_NS));
+ }
+
+ @Test
+ public void predicateLiteralsRetainNanoseconds() {
+ for (long nanos : new long[] {Long.MIN_VALUE, -1, 0, 1, Long.MAX_VALUE}) {
+ java.time.LocalDateTime value = java.time.LocalDateTime.ofEpochSecond(
+ Math.floorDiv(nanos, 1_000_000_000L), (int) Math.floorMod(nanos, 1_000_000_000L),
+ java.time.ZoneOffset.UTC);
+ org.apache.doris.datasource.paimon.source.PaimonValueConverter converter =
+ new org.apache.doris.datasource.paimon.source.PaimonValueConverter(
+ new org.apache.doris.analysis.TimeStampNsLiteral(value));
+ Assertions.assertEquals(value, converter.visit(new TimestampType(9)).toLocalDateTime());
+ }
+ }
+
+ @Test
+ public void nestedNanosecondsRequireTheLosslessReader() {
+ Assertions.assertTrue(containsNanosecondTimestamp(new TimestampType(9)));
+ Assertions.assertTrue(containsNanosecondTimestamp(
+ new ArrayType(new TimestampType(8))));
+ Assertions.assertTrue(containsNanosecondTimestamp(
+ new MapType(new IntType(), new TimestampType(7))));
+ Assertions.assertTrue(containsNanosecondTimestamp(
+ RowType.of(new IntType(), new ArrayType(new TimestampType(9)))));
+ Assertions.assertFalse(containsNanosecondTimestamp(
+ new ArrayType(new TimestampType(6))));
+ Assertions.assertFalse(containsNanosecondTimestamp(
+ new LocalZonedTimestampType(9)));
+ }
+
+ private boolean containsNanosecondTimestamp(org.apache.paimon.types.DataType type) {
+ return PaimonUtil.containsTimestampNs(PaimonUtil.paimonTypeToDorisType(type, false, false));
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
index 38b0169741e..b4777273b49 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
@@ -526,7 +526,7 @@ public class PaimonUtilTest {
@Test
public void testGeneratePartitionInfoSupportsTimestampWithoutTimeZone() {
List<Column> partitionColumns = Collections.singletonList(
- new Column("part", org.apache.doris.catalog.ScalarType.createDatetimeV2Type(6)));
+ new Column("part", org.apache.doris.catalog.Type.TIMESTAMP_NS));
DataField partitionField = DataTypes.FIELD(0, "part", DataTypes.TIMESTAMP(9));
Table table = mockPartitionTable(Collections.emptyMap(), partitionField);
BinaryRow partitionRow = new BinaryRow(1);
@@ -543,7 +543,7 @@ public class PaimonUtilTest {
partitionInfo.getPruningStatus());
PartitionItem partitionItem =
partitionInfo.getNameToPartitionItem().values().iterator().next();
- Assert.assertEquals(Collections.singletonList("2026-07-29 12:34:56.123456"),
+ Assert.assertEquals(Collections.singletonList("2026-07-29 12:34:56.123456789"),
((ListPartitionItem) partitionItem).getItems().get(0)
.getPartitionValuesAsStringList());
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinitionTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinitionTest.java
index b50d450a678..d0f1ea0193f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinitionTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinitionTest.java
@@ -17,8 +17,11 @@
package org.apache.doris.nereids.trees.plans.commands.info;
+import org.apache.doris.catalog.KeysType;
+import org.apache.doris.nereids.types.TimeStampNsType;
import org.apache.doris.nereids.types.UuidType;
+import com.google.common.collect.ImmutableSet;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -39,4 +42,21 @@ public class ColumnDefinitionTest {
Optional.of(new DefaultValue("00112233-4455-6677-8899-aabbccddeeff")), "");
Assertions.assertFalse(literal.hasUuidDefaultValue());
}
+
+ @Test
+ public void testTimestampNsDefaultsAreNormalizedForExternalDdl() {
+ String[][] values = {
+ {"20240102", "2024-01-02 00:00:00.000000000"},
+ {"20240102030405", "2024-01-02 03:04:05.000000000"},
+ {"2024-01-02 03:04:05.123456789", "2024-01-02 03:04:05.123456789"}
+ };
+ for (String[] value : values) {
+ ColumnDefinition column = new ColumnDefinition("ts", TimeStampNsType.INSTANCE,
+ false, null, true, Optional.of(new DefaultValue(value[0])), "");
+ column.validate(false, ImmutableSet.of(), ImmutableSet.of(), false, KeysType.DUP_KEYS);
+ Assertions.assertEquals(value[1], column.translateToCatalogStyle().getDefaultValue());
+ Assertions.assertEquals(value[1], column.translateToCatalogStyleForSchemaChange().getDefaultValue());
+ }
+ }
+
}
diff --git a/regression-test/data/datatype_p0/uuid/test_uuid_orc_varbinary_mapping.out b/regression-test/data/datatype_p0/uuid/test_uuid_orc_varbinary_mapping.out
index b26e248febe..fcf86545689 100644
--- a/regression-test/data/datatype_p0/uuid/test_uuid_orc_varbinary_mapping.out
+++ b/regression-test/data/datatype_p0/uuid/test_uuid_orc_varbinary_mapping.out
@@ -1,44 +1,4 @@
-- This file is automatically generated. You should know what you did if you want to edit this
--- !schema_false_false --
-a array<uuid> Yes false \N NONE
-b varbinary(65533) Yes false \N NONE
-id int Yes false \N NONE
-m map<uuid,uuid> Yes false \N NONE
-s struct<k:uuid> Yes false \N NONE
-u uuid Yes false \N NONE
-
--- !values_false_false --
-1 00112233-4455-6677-8899-aabbccddeeff ["00112233-4455-6677-8899-aabbccddeeff", null] ["00000000-0000-0000-0000-000000000000", "ffffffff-ffff-ffff-ffff-ffffffffffff"] ["80000000-0000-0000-0000-000000000000", null] 80000000-0000-0000-0000-000000000000 0000FF 6
-2 80000000-0000-0000-0000-000000000000 [] [] [] \N 0
-3 ffffffff-ffff-ffff-ffff-ffffffffffff \N \N \N \N FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 32
-4 00000000-0000-0000-0000-000000000000 [null] ["00112233-4455-6677-8899-aabbccddeeff"] ["00000000-0000-0000-0000-000000000000"] 00000000-0000-0000-0000-000000000000 \N \N
-5 \N \N \N \N \N \N \N
-
--- !filtered_false_false --
-2 80000000-0000-0000-0000-000000000000 0
-3 ffffffff-ffff-ffff-ffff-ffffffffffff FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 32
-4 00000000-0000-0000-0000-000000000000 \N \N
-
--- !schema_false_true --
-a array<uuid> Yes false \N NONE
-b varbinary(65533) Yes false \N NONE
-id int Yes false \N NONE
-m map<uuid,uuid> Yes false \N NONE
-s struct<k:uuid> Yes false \N NONE
-u uuid Yes false \N NONE
-
--- !values_false_true --
-1 00112233-4455-6677-8899-aabbccddeeff ["00112233-4455-6677-8899-aabbccddeeff", null] ["00000000-0000-0000-0000-000000000000", "ffffffff-ffff-ffff-ffff-ffffffffffff"] ["80000000-0000-0000-0000-000000000000", null] 80000000-0000-0000-0000-000000000000 0000FF 6
-2 80000000-0000-0000-0000-000000000000 [] [] [] \N 0
-3 ffffffff-ffff-ffff-ffff-ffffffffffff \N \N \N \N FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 32
-4 00000000-0000-0000-0000-000000000000 [null] ["00112233-4455-6677-8899-aabbccddeeff"] ["00000000-0000-0000-0000-000000000000"] 00000000-0000-0000-0000-000000000000 \N \N
-5 \N \N \N \N \N \N \N
-
--- !filtered_false_true --
-2 80000000-0000-0000-0000-000000000000 0
-3 ffffffff-ffff-ffff-ffff-ffffffffffff FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 32
-4 00000000-0000-0000-0000-000000000000 \N \N
-
-- !schema_true_false --
a array<uuid> Yes false \N NONE
b varbinary(65533) Yes false \N NONE
@@ -78,4 +38,3 @@ u uuid Yes false \N NONE
2 80000000-0000-0000-0000-000000000000 0
3 ffffffff-ffff-ffff-ffff-ffffffffffff FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 32
4 00000000-0000-0000-0000-000000000000 \N \N
-
diff --git a/regression-test/data/datatype_p0/uuid/test_uuid_parquet_native.out b/regression-test/data/datatype_p0/uuid/test_uuid_parquet_native.out
index 9862669c7bb..7d0d9b295a9 100644
--- a/regression-test/data/datatype_p0/uuid/test_uuid_parquet_native.out
+++ b/regression-test/data/datatype_p0/uuid/test_uuid_parquet_native.out
@@ -1,9 +1,9 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !binary_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !binary_values --
0 \N \N
@@ -21,9 +21,9 @@ s struct<k:text> Yes false \N NONE
-- !external_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !external_values --
0 \N \N \N \N \N
@@ -111,9 +111,9 @@ ffffffff-ffff-ffff-ffff-ffffffffffff 682 1396736
-- !binary_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !binary_values --
0 \N \N
@@ -131,9 +131,9 @@ s struct<k:text> Yes false \N NONE
-- !external_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !external_values --
0 \N \N \N \N \N
@@ -221,9 +221,9 @@ ffffffff-ffff-ffff-ffff-ffffffffffff 682 1396736
-- !binary_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !binary_values --
0 \N \N
@@ -241,9 +241,9 @@ s struct<k:text> Yes false \N NONE
-- !external_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !external_values --
0 \N \N \N \N \N
@@ -331,9 +331,9 @@ ffffffff-ffff-ffff-ffff-ffffffffffff 682 1396736
-- !binary_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !binary_values --
0 \N \N
@@ -351,9 +351,9 @@ s struct<k:text> Yes false \N NONE
-- !external_schema --
id int Yes false \N NONE
-u text Yes false \N NONE
-a array<text> Yes false \N NONE
-s struct<k:text> Yes false \N NONE
+u uuid Yes false \N NONE
+a array<uuid> Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
-- !external_values --
0 \N \N \N \N \N
diff --git a/regression-test/data/datatype_p0/uuid/test_uuid_parquet_roundtrip.out b/regression-test/data/datatype_p0/uuid/test_uuid_parquet_roundtrip.out
index 74b060caba8..f6d33bd28b2 100644
--- a/regression-test/data/datatype_p0/uuid/test_uuid_parquet_roundtrip.out
+++ b/regression-test/data/datatype_p0/uuid/test_uuid_parquet_roundtrip.out
@@ -10,8 +10,8 @@
3 \N \N \N
-- !inferred_types --
-a array<text> Yes false \N NONE
+a array<uuid> Yes false \N NONE
id int Yes false \N NONE
-s struct<k:text> Yes false \N NONE
-u text Yes false \N NONE
+s struct<k:uuid> Yes false \N NONE
+u uuid Yes false \N NONE
diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
index 9efb4ee587f..b58b1360a04 100644
--- a/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
+++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
@@ -1,17 +1,4 @@
-- This file is automatically generated. You should know what you did if you want to edit this
--- !v1_parquet_top_level --
-1 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-10 false 1006 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-value-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-11 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-5 \N \N \N \N \N \N \N \N \N post-null-required \N \N \N
-6 \N 606 \N \N \N \N \N \N \N post-value-required \N \N \N
-7 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-8 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-9 false \N 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-null-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-
-- !v1_parquet_physical_value_precedence --
10 1006 explicit-value-string \N \N \N
11 35 write-default \N \N \N
@@ -21,19 +8,6 @@
8 35 write-default \N \N \N
9 \N explicit-null-string \N \N \N
--- !v1_parquet_struct_children --
-1 \N \N \N \N \N \N \N \N \N \N \N \N \N
-10 \N \N \N \N \N \N \N \N \N \N \N \N \N
-11 \N \N \N \N \N \N \N \N \N \N \N \N \N
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-5 \N \N \N \N \N \N \N \N \N post-null-struct-required \N \N \N
-6 \N 706 \N \N \N \N \N \N \N post-value-struct-required \N \N \N
-7 \N \N \N \N \N \N \N \N \N \N \N \N \N
-8 \N \N \N \N \N \N \N \N \N \N \N \N \N
-9 \N \N \N \N \N \N \N \N \N \N \N \N \N
-
-- !v1_parquet_complex_parent_nulls --
1 \N \N \N \N \N \N \N
10 \N \N \N \N \N \N \N
@@ -68,25 +42,6 @@
3
4
--- !v1_parquet_legacy_mapping --
-1 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-2 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-3 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-4 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-
--- !v1_orc_top_level --
-1 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-10 false 1006 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-value-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-11 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-5 \N \N \N \N \N \N \N \N \N post-null-required \N \N \N
-6 \N 606 \N \N \N \N \N \N \N post-value-required \N \N \N
-7 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-8 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-9 false \N 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-null-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-
-- !v1_orc_physical_value_precedence --
10 1006 explicit-value-string \N \N \N
11 35 write-default \N \N \N
@@ -96,19 +51,6 @@
8 35 write-default \N \N \N
9 \N explicit-null-string \N \N \N
--- !v1_orc_struct_children --
-1 \N \N \N \N \N \N \N \N \N \N \N \N \N
-10 \N \N \N \N \N \N \N \N \N \N \N \N \N
-11 \N \N \N \N \N \N \N \N \N \N \N \N \N
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-5 \N \N \N \N \N \N \N \N \N post-null-struct-required \N \N \N
-6 \N 706 \N \N \N \N \N \N \N post-value-struct-required \N \N \N
-7 \N \N \N \N \N \N \N \N \N \N \N \N \N
-8 \N \N \N \N \N \N \N \N \N \N \N \N \N
-9 \N \N \N \N \N \N \N \N \N \N \N \N \N
-
-- !v1_orc_complex_parent_nulls --
1 \N \N \N \N \N \N \N
10 \N \N \N \N \N \N \N
@@ -143,24 +85,18 @@
3
4
--- !v1_orc_legacy_mapping --
-1 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-2 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-3 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-4 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-
-- !v2_parquet_top_level --
-1 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-10 false 1006 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-value-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-11 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
+1 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+10 false 1006 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-value-string 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+11 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
5 \N \N \N \N \N \N \N \N \N post-null-required \N \N \N
6 \N 606 \N \N \N \N \N \N \N post-value-required \N \N \N
-7 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-8 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-9 false \N 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-null-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
+7 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+8 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+9 false \N 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-null-string 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
-- !v2_parquet_physical_value_precedence --
10 1006 explicit-value-string \N \N \N
@@ -175,9 +111,9 @@
1 \N \N \N \N \N \N \N \N \N \N \N \N \N
10 \N \N \N \N \N \N \N \N \N \N \N \N \N
11 \N \N \N \N \N \N \N \N \N \N \N \N \N
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
+2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
5 \N \N \N \N \N \N \N \N \N post-null-struct-required \N \N \N
6 \N 706 \N \N \N \N \N \N \N post-value-struct-required \N \N \N
7 \N \N \N \N \N \N \N \N \N \N \N \N \N
@@ -219,23 +155,23 @@
4
-- !v2_parquet_legacy_mapping --
-1 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-2 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-3 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-4 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+1 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+2 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+3 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+4 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-- !v2_orc_top_level --
-1 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-10 false 1006 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-value-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-11 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
+1 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+10 false 1006 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-value-string 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+11 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
5 \N \N \N \N \N \N \N \N \N post-null-required \N \N \N
6 \N 606 \N \N \N \N \N \N \N post-value-required \N \N \N
-7 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-8 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
-9 false \N 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-null-string 123E4567E89B12D3A456426614174001 1A1B1C1D 3A3B3C
+7 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+8 false 35 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 write-default 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
+9 false \N 4900000001 13.5 456.75 98765.4321 2025-01-18 2025-01-18 01:02:03.654321 2025-01-18 01:02:03.654321+00:00 explicit-null-string 123e4567-e89b-12d3-a456-426614174001 1A1B1C1D 3A3B3C
-- !v2_orc_physical_value_precedence --
10 1006 explicit-value-string \N \N \N
@@ -250,9 +186,9 @@
1 \N \N \N \N \N \N \N \N \N \N \N \N \N
10 \N \N \N \N \N \N \N \N \N \N \N \N \N
11 \N \N \N \N \N \N \N \N \N \N \N \N \N
-2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
-4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C
+2 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+3 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
+4 true 34 4900000000 12.25 -123.5 12345.6789 2024-12-17 2024-12-17 23:59:59.123456 2024-12-17 23:59:59.123456+00:00 initial-default 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
5 \N \N \N \N \N \N \N \N \N post-null-struct-required \N \N \N
6 \N 706 \N \N \N \N \N \N \N post-value-struct-required \N \N \N
7 \N \N \N \N \N \N \N \N \N \N \N \N \N
@@ -294,10 +230,10 @@
4
-- !v2_orc_legacy_mapping --
-1 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-2 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-3 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-4 123E4567E89B12D3A456426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+1 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+2 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+3 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
+4 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C 2024-12-17 23:59:59.123456+00:00
-- !parquet_branch_overwrite_default --
14 36
diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_timestamp_ns.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_timestamp_ns.out
new file mode 100644
index 00000000000..d91b1cbab50
--- /dev/null
+++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_timestamp_ns.out
@@ -0,0 +1,383 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !parquet_schema --
+id int Yes true \N
+ts timestamp_ns Yes true \N
+items array<timestamp_ns> Yes true \N
+detail struct<t:timestamp_ns> Yes true \N
+entries map<text,timestamp_ns> Yes true \N
+
+-- !parquet_tvf_schema --
+id int Yes false \N NONE
+ts timestamp_ns Yes false \N NONE
+items array<timestamp_ns> Yes false \N NONE
+detail struct<t:timestamp_ns> Yes false \N NONE
+entries map<text,timestamp_ns> Yes false \N NONE
+
+-- !parquet_UTC_values --
+1 2024-01-02 03:04:05.123456789 ["2024-01-02 03:04:05.123456789", null] {"t":"1969-12-31 23:59:58.999999999"} {"x":"1970-01-01 00:00:00.000000001"}
+2 2024-01-02 03:04:05.123456788 [] \N {}
+3 1969-12-31 23:59:58.999999999 \N \N \N
+4 1970-01-01 00:00:00.000000001 \N \N \N
+5 \N \N \N \N
+6 1677-09-21 00:12:43.145224192 \N \N \N
+7 2262-04-11 23:47:16.854775807 \N \N \N
+
+-- !parquet_UTC_eq --
+1
+
+-- !parquet_UTC_range --
+2
+3
+4
+6
+
+-- !parquet_UTC_aggregate --
+1677-09-21 00:12:43.145224192 2262-04-11 23:47:16.854775807 6
+
+-- !parquet_Asia_Shanghai_values --
+1 2024-01-02 03:04:05.123456789 ["2024-01-02 03:04:05.123456789", null] {"t":"1969-12-31 23:59:58.999999999"} {"x":"1970-01-01 00:00:00.000000001"}
+2 2024-01-02 03:04:05.123456788 [] \N {}
+3 1969-12-31 23:59:58.999999999 \N \N \N
+4 1970-01-01 00:00:00.000000001 \N \N \N
+5 \N \N \N \N
+6 1677-09-21 00:12:43.145224192 \N \N \N
+7 2262-04-11 23:47:16.854775807 \N \N \N
+
+-- !parquet_Asia_Shanghai_eq --
+1
+
+-- !parquet_Asia_Shanghai_range --
+2
+3
+4
+6
+
+-- !parquet_Asia_Shanghai_aggregate --
+1677-09-21 00:12:43.145224192 2262-04-11 23:47:16.854775807 6
+
+-- !parquet_copy --
+1 2024-01-02 03:04:05.123456789 ["2024-01-02 03:04:05.123456789", null] {"t":"1969-12-31 23:59:58.999999999"} {"x":"1970-01-01 00:00:00.000000001"}
+2 2024-01-02 03:04:05.123456788 [] \N {}
+3 1969-12-31 23:59:58.999999999 \N \N \N
+4 1970-01-01 00:00:00.000000001 \N \N \N
+5 \N \N \N \N
+6 1677-09-21 00:12:43.145224192 \N \N \N
+7 2262-04-11 23:47:16.854775807 \N \N \N
+
+-- !parquet_defaults --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+
+-- !parquet_compact_defaults --
+1 2024-01-02 00:00:00.000000000 2024-01-02 03:04:05.000000000
+2 2024-01-02 00:00:00.000000000 2024-01-02 03:04:05.000000000
+3 2024-01-02 00:00:00.000000000 2024-01-02 03:04:05.000000000
+
+-- !parquet_identity --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_identity_overwrite --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_identity_static_date --
+2024-01-01 00:00:00.000000000
+
+-- !parquet_identity_static_offset --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+8 2024-01-01 00:00:00.000000000
+10 2023-12-31 16:00:00.000000000
+
+-- !parquet_year --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_year_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_month --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_month_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_day --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_day_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_hour --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_hour_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_bucket --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !parquet_bucket_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_schema --
+id int Yes true \N
+ts timestamp_ns Yes true \N
+items array<timestamp_ns> Yes true \N
+detail struct<t:timestamp_ns> Yes true \N
+entries map<text,timestamp_ns> Yes true \N
+
+-- !orc_tvf_schema --
+id int Yes false \N NONE
+ts timestamp_ns Yes false \N NONE
+items array<timestamp_ns> Yes false \N NONE
+detail struct<t:timestamp_ns> Yes false \N NONE
+entries map<text,timestamp_ns> Yes false \N NONE
+
+-- !orc_UTC_values --
+1 2024-01-02 03:04:05.123456789 ["2024-01-02 03:04:05.123456789", null] {"t":"1969-12-31 23:59:58.999999999"} {"x":"1970-01-01 00:00:00.000000001"}
+2 2024-01-02 03:04:05.123456788 [] \N {}
+3 1969-12-31 23:59:58.999999999 \N \N \N
+4 1970-01-01 00:00:00.000000001 \N \N \N
+5 \N \N \N \N
+6 1677-09-21 00:12:43.145224192 \N \N \N
+7 2262-04-11 23:47:16.854775807 \N \N \N
+
+-- !orc_UTC_eq --
+1
+
+-- !orc_UTC_range --
+2
+3
+4
+6
+
+-- !orc_UTC_aggregate --
+1677-09-21 00:12:43.145224192 2262-04-11 23:47:16.854775807 6
+
+-- !orc_Asia_Shanghai_values --
+1 2024-01-02 03:04:05.123456789 ["2024-01-02 03:04:05.123456789", null] {"t":"1969-12-31 23:59:58.999999999"} {"x":"1970-01-01 00:00:00.000000001"}
+2 2024-01-02 03:04:05.123456788 [] \N {}
+3 1969-12-31 23:59:58.999999999 \N \N \N
+4 1970-01-01 00:00:00.000000001 \N \N \N
+5 \N \N \N \N
+6 1677-09-21 00:12:43.145224192 \N \N \N
+7 2262-04-11 23:47:16.854775807 \N \N \N
+
+-- !orc_Asia_Shanghai_eq --
+1
+
+-- !orc_Asia_Shanghai_range --
+2
+3
+4
+6
+
+-- !orc_Asia_Shanghai_aggregate --
+1677-09-21 00:12:43.145224192 2262-04-11 23:47:16.854775807 6
+
+-- !orc_copy --
+1 2024-01-02 03:04:05.123456789 ["2024-01-02 03:04:05.123456789", null] {"t":"1969-12-31 23:59:58.999999999"} {"x":"1970-01-01 00:00:00.000000001"}
+2 2024-01-02 03:04:05.123456788 [] \N {}
+3 1969-12-31 23:59:58.999999999 \N \N \N
+4 1970-01-01 00:00:00.000000001 \N \N \N
+5 \N \N \N \N
+6 1677-09-21 00:12:43.145224192 \N \N \N
+7 2262-04-11 23:47:16.854775807 \N \N \N
+
+-- !orc_defaults --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+
+-- !orc_compact_defaults --
+1 2024-01-02 00:00:00.000000000 2024-01-02 03:04:05.000000000
+2 2024-01-02 00:00:00.000000000 2024-01-02 03:04:05.000000000
+3 2024-01-02 00:00:00.000000000 2024-01-02 03:04:05.000000000
+
+-- !orc_identity --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_identity_overwrite --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_identity_static_date --
+2024-01-01 00:00:00.000000000
+
+-- !orc_identity_static_offset --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+8 2024-01-01 00:00:00.000000000
+10 2023-12-31 16:00:00.000000000
+
+-- !orc_year --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_year_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_month --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_month_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_day --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_day_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_hour --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_hour_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_bucket --
+1 2024-01-02 03:04:05.123456789
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
+-- !orc_bucket_overwrite --
+2 2024-01-02 03:04:05.123456788
+3 1969-12-31 23:59:58.999999999
+4 1970-01-01 00:00:00.000000001
+5 \N
+6 1677-09-21 00:12:43.145224192
+7 2262-04-11 23:47:16.854775807
+
diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_uuid_binary_compatibility.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_uuid_binary_compatibility.out
index 50d6b1e9a47..4f60f8dc67c 100644
--- a/regression-test/data/external_table_p0/iceberg/test_iceberg_uuid_binary_compatibility.out
+++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_uuid_binary_compatibility.out
@@ -1,94 +1,44 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !uuid_write_parquet_false_insert --
-1 00000000000000000000000000000000
-2 80000000000000000000000000000000
-3 00112233445566778899AABBCCDDEEFF
+1 00000000-0000-0000-0000-000000000000
+2 80000000-0000-0000-0000-000000000000
+3 00112233-4455-6677-8899-aabbccddeeff
4 \N
-- !uuid_write_parquet_false_overwrite --
-11 00112233445566778899AABBCCDDEEFF
-12 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+11 00112233-4455-6677-8899-aabbccddeeff
+12 ffffffff-ffff-ffff-ffff-ffffffffffff
13 \N
-- !uuid_write_orc_false_insert --
-1 00000000000000000000000000000000
-2 80000000000000000000000000000000
-3 00112233445566778899AABBCCDDEEFF
+1 00000000-0000-0000-0000-000000000000
+2 80000000-0000-0000-0000-000000000000
+3 00112233-4455-6677-8899-aabbccddeeff
4 \N
-- !uuid_write_orc_false_overwrite --
-11 00112233445566778899AABBCCDDEEFF
-12 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+11 00112233-4455-6677-8899-aabbccddeeff
+12 ffffffff-ffff-ffff-ffff-ffffffffffff
13 \N
--- !uuid_parquet_parquet_binary_false_v2_false_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_false_predicate --
-4
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_false_predicate --
-4
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
-6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_true_predicate --
-4
-
--- !uuid_parquet_parquet_binary_false_v2_false_strict_true_predicate --
-4
-
-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_ids --
3
@@ -97,6 +47,9 @@
-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_count --
2
+-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_parquet_parquet_binary_false_v2_true_strict_false_predicate --
4
@@ -104,23 +57,23 @@
4
-- !uuid_parquet_orc_binary_false_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_false_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_false_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_orc_binary_false_v2_true_strict_false_ids --
3
@@ -129,6 +82,9 @@
-- !uuid_parquet_orc_binary_false_v2_true_strict_false_count --
2
+-- !uuid_parquet_orc_binary_false_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_parquet_orc_binary_false_v2_true_strict_false_predicate --
4
@@ -136,23 +92,23 @@
4
-- !uuid_orc_parquet_binary_false_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_false_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_false_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_parquet_binary_false_v2_true_strict_false_ids --
3
@@ -161,6 +117,9 @@
-- !uuid_orc_parquet_binary_false_v2_true_strict_false_count --
2
+-- !uuid_orc_parquet_binary_false_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_orc_parquet_binary_false_v2_true_strict_false_predicate --
4
@@ -168,23 +127,23 @@
4
-- !uuid_orc_orc_binary_false_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_false_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_false_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_orc_binary_false_v2_true_strict_false_ids --
3
@@ -193,6 +152,9 @@
-- !uuid_orc_orc_binary_false_v2_true_strict_false_count --
2
+-- !uuid_orc_orc_binary_false_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_orc_orc_binary_false_v2_true_strict_false_predicate --
4
@@ -200,23 +162,23 @@
4
-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_ids --
3
@@ -225,6 +187,9 @@
-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_count --
2
+-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_parquet_parquet_binary_false_v2_true_strict_true_predicate --
4
@@ -232,23 +197,23 @@
4
-- !uuid_parquet_orc_binary_false_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_false_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_false_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_orc_binary_false_v2_true_strict_true_ids --
3
@@ -257,6 +222,9 @@
-- !uuid_parquet_orc_binary_false_v2_true_strict_true_count --
2
+-- !uuid_parquet_orc_binary_false_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_parquet_orc_binary_false_v2_true_strict_true_predicate --
4
@@ -264,23 +232,23 @@
4
-- !uuid_orc_parquet_binary_false_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_false_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_false_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_parquet_binary_false_v2_true_strict_true_ids --
3
@@ -289,6 +257,9 @@
-- !uuid_orc_parquet_binary_false_v2_true_strict_true_count --
2
+-- !uuid_orc_parquet_binary_false_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_orc_parquet_binary_false_v2_true_strict_true_predicate --
4
@@ -296,23 +267,23 @@
4
-- !uuid_orc_orc_binary_false_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_false_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_false_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_orc_binary_false_v2_true_strict_true_ids --
3
@@ -321,6 +292,9 @@
-- !uuid_orc_orc_binary_false_v2_true_strict_true_count --
2
+-- !uuid_orc_orc_binary_false_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_orc_orc_binary_false_v2_true_strict_true_predicate --
4
@@ -328,45 +302,45 @@
4
-- !uuid_write_parquet_true_insert --
-1 00000000000000000000000000000000
-2 80000000000000000000000000000000
-3 00112233445566778899AABBCCDDEEFF
+1 00000000-0000-0000-0000-000000000000
+2 80000000-0000-0000-0000-000000000000
+3 00112233-4455-6677-8899-aabbccddeeff
4 \N
-- !uuid_write_parquet_true_overwrite --
-11 00112233445566778899AABBCCDDEEFF
-12 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+11 00112233-4455-6677-8899-aabbccddeeff
+12 ffffffff-ffff-ffff-ffff-ffffffffffff
13 \N
-- !uuid_write_orc_true_insert --
-1 00000000000000000000000000000000
-2 80000000000000000000000000000000
-3 00112233445566778899AABBCCDDEEFF
+1 00000000-0000-0000-0000-000000000000
+2 80000000-0000-0000-0000-000000000000
+3 00112233-4455-6677-8899-aabbccddeeff
4 \N
-- !uuid_write_orc_true_overwrite --
-11 00112233445566778899AABBCCDDEEFF
-12 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+11 00112233-4455-6677-8899-aabbccddeeff
+12 ffffffff-ffff-ffff-ffff-ffffffffffff
13 \N
-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_ids --
3
@@ -375,6 +349,9 @@
-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_count --
2
+-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_parquet_parquet_binary_true_v2_true_strict_false_predicate --
4
@@ -382,23 +359,23 @@
4
-- !uuid_parquet_orc_binary_true_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_true_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_true_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_orc_binary_true_v2_true_strict_false_ids --
3
@@ -407,6 +384,9 @@
-- !uuid_parquet_orc_binary_true_v2_true_strict_false_count --
2
+-- !uuid_parquet_orc_binary_true_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_parquet_orc_binary_true_v2_true_strict_false_predicate --
4
@@ -414,23 +394,23 @@
4
-- !uuid_orc_parquet_binary_true_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_true_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_true_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_parquet_binary_true_v2_true_strict_false_ids --
3
@@ -439,6 +419,9 @@
-- !uuid_orc_parquet_binary_true_v2_true_strict_false_count --
2
+-- !uuid_orc_parquet_binary_true_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_orc_parquet_binary_true_v2_true_strict_false_predicate --
4
@@ -446,23 +429,23 @@
4
-- !uuid_orc_orc_binary_true_v2_true_strict_false_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_true_v2_true_strict_false_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_true_v2_true_strict_false_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_orc_binary_true_v2_true_strict_false_ids --
3
@@ -471,6 +454,9 @@
-- !uuid_orc_orc_binary_true_v2_true_strict_false_count --
2
+-- !uuid_orc_orc_binary_true_v2_true_strict_false_predicate_reference --
+4
+
-- !uuid_orc_orc_binary_true_v2_true_strict_false_predicate --
4
@@ -478,23 +464,23 @@
4
-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_ids --
3
@@ -503,6 +489,9 @@
-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_count --
2
+-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_parquet_parquet_binary_true_v2_true_strict_true_predicate --
4
@@ -510,23 +499,23 @@
4
-- !uuid_parquet_orc_binary_true_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_true_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_parquet_orc_binary_true_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_parquet_orc_binary_true_v2_true_strict_true_ids --
3
@@ -535,6 +524,9 @@
-- !uuid_parquet_orc_binary_true_v2_true_strict_true_count --
2
+-- !uuid_parquet_orc_binary_true_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_parquet_orc_binary_true_v2_true_strict_true_predicate --
4
@@ -542,23 +534,23 @@
4
-- !uuid_orc_parquet_binary_true_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_true_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_parquet_binary_true_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_parquet_binary_true_v2_true_strict_true_ids --
3
@@ -567,6 +559,9 @@
-- !uuid_orc_parquet_binary_true_v2_true_strict_true_count --
2
+-- !uuid_orc_parquet_binary_true_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_orc_parquet_binary_true_v2_true_strict_true_predicate --
4
@@ -574,23 +569,23 @@
4
-- !uuid_orc_orc_binary_true_v2_true_strict_true_before --
-0 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-1 00112233445566778899AABBCCDDEEFF \N
-2 00112233445566778899AABBCCDDEEFF 00112233445566778899AABBCCDDEEFF
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-5 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
+0 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+1 00112233-4455-6677-8899-aabbccddeeff \N
+2 00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+5 ffffffff-ffff-ffff-ffff-ffffffffffff ffffffff-ffff-ffff-ffff-ffffffffffff
6 \N \N
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_true_v2_true_strict_true_equality_only --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
-7 6162636465666768696A6B6C6D6E6F70 6162636465666768696A6B6C6D6E6F70
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
+7 61626364-6566-6768-696a-6b6c6d6e6f70 61626364-6566-6768-696a-6b6c6d6e6f70
-- !uuid_orc_orc_binary_true_v2_true_strict_true_deleted --
-3 00000000000000000000000000000000 00000000000000000000000000000000
-4 80000000000000000000000000000000 80000000000000000000000000000000
+3 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000
+4 80000000-0000-0000-0000-000000000000 80000000-0000-0000-0000-000000000000
-- !uuid_orc_orc_binary_true_v2_true_strict_true_ids --
3
@@ -599,6 +594,9 @@
-- !uuid_orc_orc_binary_true_v2_true_strict_true_count --
2
+-- !uuid_orc_orc_binary_true_v2_true_strict_true_predicate_reference --
+4
+
-- !uuid_orc_orc_binary_true_v2_true_strict_true_predicate --
4
diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_uuid_inputs.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_uuid_inputs.out
... 2441 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org