You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huan233usc (via GitHub)" <gi...@apache.org> on 2026/06/11 20:38:46 UTC
[PR] feat: support column default values (initial-default / write-default) [iceberg-cpp]
huan233usc opened a new pull request, #731:
URL: https://github.com/apache/iceberg-cpp/pull/731
Closes #730 (item 2 of #637).
Implements Iceberg v3 column default values: `initial-default` / `write-default` on the
schema, JSON serde, read-path application, schema-evolution support, and format-version
validation.
## What changed
### Schema model
- `SchemaField` carries optional `initial_default` / `write_default` literals
(`std::shared_ptr<Literal>` to keep `schema_field.h` free of the
`literal.h → type.h → schema_field.h` include cycle), with `WithInitialDefault` /
`WithWriteDefault` copy-modifiers in the style of `AsRequired`/`AsOptional`.
- `SchemaField::Validate()` checks that defaults are primitive literals matching the
field type; `Schema::Validate(format_version)` rejects schemas with defaults below v3
(uses the previously-unused `TableMetadata::kMinFormatVersionDefaultValues`, resolving
the TODO there).
### JSON serde
- `FieldFromJson` / `ToJson(SchemaField)` parse and write `initial-default` /
`write-default` using the existing single-value serialization
(`LiteralFromJson(json, type)`), resolving the `add default values` TODO in struct
serialization. All primitive types supported (incl. decimal, fixed, uuid, temporal).
### Read path (`initial-default`)
- `Project()` maps a column missing from a data file to
`FieldProjection::Kind::kDefault` carrying the literal when the field has an
`initial-default` — for required *and* optional fields, per the spec ("optional with
default" reads the default, not null). Resolves the default-value TODO in
`schema_util.cc`; the Avro-side projection in `avro_schema_util.cc` gets the same
branch.
- New `iceberg::arrow` helpers (`literal_util`) convert a `Literal` to an Arrow scalar /
constant array; the Parquet reader materializes `kDefault` via `MakeDefaultArray` and
the Avro reader via `AppendDefaultToBuilder`.
### Schema evolution (`write-default`)
- `AddColumn` / `AddRequiredColumn` accept an optional `default_value`, used as both the
`initial-default` and `write-default` of the new column (Java parity). A required
column with a default no longer needs `AllowIncompatibleChanges()`.
- `RequireColumn()` accepts a column added with a default in the same update (resolves
the defaulted-add TODO in `UpdateColumnRequirementInternal`).
- New `UpdateColumnDefault()` updates the `write-default` of an existing column
(`initial-default` stays fixed once the column exists).
- `UpdateColumnDoc` / `RenameColumn` / `UpdateColumn` preserve defaults when
reconstructing the field; type promotion casts the defaults to the new type.
### Scope note: write-path application
Writers in this library consume complete Arrow arrays, so filling omitted columns with
`write-default` at write time remains the engine's responsibility, as in Java. The
library's role — storing, validating, serializing the defaults, and exposing them
through schema evolution — is covered here.
## Testing
- Schema serde round-trips (top-level + nested struct fields, mismatch rejection).
- `Schema::Validate`: v2 rejects defaults, v3 accepts; mismatched default type rejected.
- Projection: missing required/optional fields with `initial-default` → `kDefault`;
present fields ignore `initial-default`.
- Parquet `ProjectRecordBatch` and Avro `AppendDatumToBuilder`: missing columns
materialize the default at top level and in nested structs.
- `UpdateSchema`: add with default (both defaults set), required-with-default without
`AllowIncompatibleChanges()`, mismatched default rejected, `UpdateColumnDefault`,
`RequireColumn` on defaulted add, doc-update preservation, type-promotion casting,
and v2 rejection at `Apply()`; new `TableMetadataV3Valid.json` test resource.
- Full suite passing locally (the pre-existing S3 `file_io_test` is unrelated and fails
only in my local environment due to a Homebrew AWS-SDK ABI issue; not touched by this
change).
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [wip]feat: support column default values (initial-default / write-default) [iceberg-cpp]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on PR #731:
URL: https://github.com/apache/iceberg-cpp/pull/731#issuecomment-5200242970
Yes closing this one and will look into the remaning of write path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [wip]feat: support column default values (initial-default / write-default) [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #731:
URL: https://github.com/apache/iceberg-cpp/pull/731#issuecomment-5163078154
IIUC, this PR can be closed, right? It seems that the only remaining work of https://github.com/apache/iceberg-cpp/issues/730 is the write path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [wip]feat: support column default values (initial-default / write-default) [iceberg-cpp]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc closed pull request #731: [wip]feat: support column default values (initial-default / write-default)
URL: https://github.com/apache/iceberg-cpp/pull/731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [wip]feat: support column default values (initial-default / write-default) [iceberg-cpp]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #731:
URL: https://github.com/apache/iceberg-cpp/pull/731#discussion_r3401514324
##########
src/iceberg/schema_field.cc:
##########
@@ -55,13 +58,82 @@ bool SchemaField::optional() const { return optional_; }
std::string_view SchemaField::doc() const { return doc_; }
+std::optional<std::reference_wrapper<const Literal>> SchemaField::initial_default()
+ const {
+ if (initial_default_ == nullptr) {
+ return std::nullopt;
+ }
+ return std::cref(*initial_default_);
+}
+
+std::optional<std::reference_wrapper<const Literal>> SchemaField::write_default() const {
+ if (write_default_ == nullptr) {
+ return std::nullopt;
+ }
+ return std::cref(*write_default_);
+}
+
+SchemaField SchemaField::WithInitialDefault(Literal initial_default) const {
Review Comment:
Alternative to those with methods are make it in constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [wip]feat: support column default values (initial-default / write-default) [iceberg-cpp]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on PR #731:
URL: https://github.com/apache/iceberg-cpp/pull/731#issuecomment-4710594471
Splitting this into smaller reviewable PRs; keeping this one open as the end-to-end proof-of-concept. Part 1/4 (schema foundation): #746. Read-path (Parquet, then Avro) and schema-evolution parts will follow, stacked on #746.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org