You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "twuebi (via GitHub)" <gi...@apache.org> on 2026/08/05 20:50:44 UTC

[PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

twuebi opened a new pull request, #1665:
URL: https://github.com/apache/iceberg-go/pull/1665

   fixes #1664 
   
   Decoding rejected source-id 0 in partition specs and sort orders, but positivity is only an invariant for specs already bound to a schema. Create-table and commit requests carry unbound specs whose source IDs are the client's ordinal placeholders: Spark numbers the root struct's fields from zero, so partitioning or sorting by a table's first column sends source-id 0. Those requests failed with "source ID must be positive: 0".
   
   Reject only negative IDs during decoding and leave positivity to schema binding, which is where Java enforces it. TableMetadata.newTableMetadata assigns fresh IDs and remaps each spec field by looking the source name up in the request schema; reassignIDs already does the same, so ordinal IDs were handled correctly once past decoding. Non-resolvable IDs are still caught: AddPartitionFieldBySourceID and SortOrder.CheckCompatibility report source IDs missing from the schema.
   
   Regression tests cover a create-table request numbered from zero through NewMetadata and a metadata round trip, for both partition specs and sort orders.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3750113617


##########
table/metadata_internal_test.go:
##########
@@ -711,6 +711,83 @@ func TestRejectStructurallyInvalidHistoricalPartitionSpec(t *testing.T) {
 	assert.ErrorContains(t, err, "spec ID must be non-negative")
 }
 
+// A create-table request carries unbound placeholder IDs rather than final
+// field IDs. Spark numbers the root struct's fields by ordinal, so the first
+// column is field-id 0 and partitioning or sorting by it yields source-id 0.
+// NewMetadata must accept that and remap every source ID by name, matching
+// Java's TableMetadata.newTableMetadata.
+func TestNewMetadataFromOrdinalNumberedRequest(t *testing.T) {
+	const requestSchema = `{
+		"type": "struct", "schema-id": 0,
+		"fields": [
+			{"id": 0, "name": "my_ints", "required": false, "type": "int"},
+			{"id": 1, "name": "my_floats", "required": false, "type": "double"},
+			{"id": 2, "name": "strings", "required": false, "type": "string"}
+		]
+	}`
+
+	for _, tt := range []struct {
+		name         string
+		spec         string
+		wantSourceID int
+		wantName     string
+	}{
+		{
+			name:         "identity on first column",
+			spec:         `{"spec-id":0,"fields":[{"name":"my_ints","transform":"identity","source-id":0,"field-id":1000}]}`,
+			wantSourceID: 1,
+			wantName:     "my_ints",
+		},
+		{
+			name:         "bucket on first column",
+			spec:         `{"spec-id":0,"fields":[{"name":"my_ints_bucket","transform":"bucket[16]","source-id":0,"field-id":1000}]}`,
+			wantSourceID: 1,
+			wantName:     "my_ints_bucket",
+		},
+		{
+			name:         "identity on last column",
+			spec:         `{"spec-id":0,"fields":[{"name":"strings","transform":"identity","source-id":2,"field-id":1000}]}`,
+			wantSourceID: 3,
+			wantName:     "strings",
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			var sc iceberg.Schema
+			require.NoError(t, json.Unmarshal([]byte(requestSchema), &sc))
+
+			var spec iceberg.PartitionSpec
+			require.NoError(t, json.Unmarshal([]byte(tt.spec), &spec))
+
+			var order SortOrder
+			require.NoError(t, json.Unmarshal([]byte(
+				`{"order-id":1,"fields":[{"transform":"identity","source-id":0,"direction":"asc","null-order":"nulls-first"}]}`,
+			), &order))
+
+			meta, err := NewMetadata(&sc, &spec, order, "s3://bucket/tbl", nil)
+			require.NoError(t, err)
+
+			gotSpec := meta.PartitionSpec()
+			require.Equal(t, 1, gotSpec.NumFields())
+
+			assert.Equal(t, tt.wantSourceID, gotSpec.Field(0).SourceID())
+			assert.Equal(t, tt.wantName, gotSpec.Field(0).Name)
+
+			// The sort order references the first column, so it remaps to 1.
+			for _, field := range meta.SortOrder().Fields() {

Review Comment:
   fixed, `require.Equal(t, 1, meta.SortOrder().Len())` now checks that there's something to loop



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3750103021


##########
table/sorting.go:
##########
@@ -191,9 +191,11 @@ func (s *SortField) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
+// Zero is valid: unbound sort orders carry client ordinal placeholder IDs that
+// start at 0. CheckCompatibility rejects IDs that don't resolve against a schema.
 func validateSortSourceID(id int) error {
-	if id <= 0 {
-		return fmt.Errorf("source ID must be positive: %d", id)
+	if id < 0 {

Review Comment:
   NewSortOrder rejects source-id 0 again and TestNewSortOrderRejectsZeroSourceID is restored



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3796132776


##########
partitions.go:
##########
@@ -190,6 +206,24 @@ type PartitionSpec struct {
 	sourceIdToFields map[int][]PartitionField
 }
 
+// UnboundPartitionSpec decodes a partition spec that a client sent in a
+// create-table request, before it has been bound to a schema. Such a spec
+// carries the client's placeholder source IDs rather than schema field IDs, and
+// those placeholders start at zero: Spark numbers the columns of a new table
+// from zero, so partitioning by the first column arrives as source-id 0.
+// Binding the embedded spec to a schema resolves the placeholders to field IDs.
+//
+// Use PartitionSpec for specs read from table metadata, where source IDs are
+// bound field IDs and must be positive. Catalog implementations that serve the
+// REST create-table request should decode its partition spec into this type.
+type UnboundPartitionSpec struct {

Review Comment:
   The schema argument isn't the target schema, it's the schema the source IDs refer to. For an unbound spec that's the request schema, whose field IDs are the placeholders. I probed it: source-id 0 against a request schema with fields 0,1,2 resolves cleanly to ints, no error. So `unboundSpec.BindToSchema(requestSchema, ...)` is the correct call and works today.
   
   Rust agrees, and so does Java. `UnboundPartitionSpec::bind` (`crates/iceberg/src/spec/partition.rs:275`) resolves by source ID, and `UnboundPartitionField` (`partition.rs:243-254`) carries `source_id`, `field_id`, `name`, `transform`. Java is the same: `UnboundPartitionSpec.bind` calls `schema.findType(field.sourceId)` (`UnboundPartitionSpec.java:61`) and `builder.add(field.sourceId, ...)` (line 69), and its `UnboundPartitionField` holds no source column name either. So the name isn't in the wire format at all, `name` there is the partition field name, and a name-based override isn't possible without also passing the request schema, which is what `reassignIDs` already does at `table/metadata.go:2958`.
   
   The real hazard was quieter than a failed lookup: `AssignFreshSchemaIDs` numbers from 1 (`schema.go:1554-1562`) while placeholders start at 0, so binding against the reassigned schema resolved every field off-by-one while keeping the client's name, and nothing caught it, `addSpecFieldInternal` checked only the transform's own parameters and duplicate partition names. Java does catch it: `checkAndAddPartitionName` (`PartitionSpec.java:401-427`) requires a partition name that is also a schema column to be sourced from that column, and `TableMetadata.java:1189` uses that checked bind in the same position as our `table/metadata.go:450`. I've now added the same rule in `validatePartitionNameAgainstSchema` (`partitions.go:402`), called from `addSpecFieldInternal`, which now takes the schema and so covers both `BindToSchema` and the create-table path through `reassignIDs`. The shift above now reports partition name floats matches schema column with field ID 2, but the field is sourced fr
 om 1. The only test that broke in the tree was the one asserting the silent misbind, every catalog suite and the metadata replay tests pass untouched and `TestPartitionNameMatchingSchemaColumn`(`partitions_test.go:1000`) covers the rule directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3740296753


##########
table/sorting.go:
##########
@@ -191,9 +191,11 @@ func (s *SortField) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
+// Zero is valid: unbound sort orders carry client ordinal placeholder IDs that
+// start at 0. CheckCompatibility rejects IDs that don't resolve against a schema.
 func validateSortSourceID(id int) error {
-	if id <= 0 {
-		return fmt.Errorf("source ID must be positive: %d", id)
+	if id < 0 {

Review Comment:
   Beyond the shared decode issue, relaxing `validateSortSourceID` also changes the public `NewSortOrder` constructor, which builds bound orders. It's called from there with `validateSourceIDs=true`, so a programmatic caller can now construct a `SortOrder` with source-id 0 and get no error at build time, and the flipped test (`require.Error` to `require.NoError` in `TestNewSortOrderAcceptsZeroSourceID`) turns that into a semver-observable removal of a published guarantee.
   
   I'd keep `NewSortOrder` rejecting source-id 0 and let only the request-decode path permit it. Happy to be wrong if there's a bound case where 0 is legitimate, but I can't think of one.



##########
table/metadata_internal_test.go:
##########
@@ -711,6 +711,83 @@ func TestRejectStructurallyInvalidHistoricalPartitionSpec(t *testing.T) {
 	assert.ErrorContains(t, err, "spec ID must be non-negative")
 }
 
+// A create-table request carries unbound placeholder IDs rather than final
+// field IDs. Spark numbers the root struct's fields by ordinal, so the first
+// column is field-id 0 and partitioning or sorting by it yields source-id 0.
+// NewMetadata must accept that and remap every source ID by name, matching
+// Java's TableMetadata.newTableMetadata.
+func TestNewMetadataFromOrdinalNumberedRequest(t *testing.T) {

Review Comment:
   This covers the accept side nicely, and the round-trip assert is a good touch. The reject side zeroshade asked for isn't here yet though, and there's a real gap behind it: `checkSortOrders` only runs `CheckCompatibility` on the default order and `checkPartitionSpecs` only checks that the default spec id appears in the list. So once the parser stops rejecting source-id 0, a persisted historical (non-default) spec or order carrying 0 has no validation net anywhere.
   
   I'd add persisted v1/v2/v3 rejection tests here covering historical specs/orders, not just the default, plus a round-trip check that write paths only ever emit positive IDs. If instead the permissive parse path stays, `checkSortOrders` and `checkPartitionSpecs` would need to validate every spec/order rather than just the default.



##########
partitions.go:
##########
@@ -162,10 +162,13 @@ func (p *PartitionField) UnmarshalJSON(b []byte) error {
 	} else {
 		p.SourceIDs = []int{aux.SourceID}
 	}
+	// Positivity only holds for specs already bound to a schema. Decoding also
+	// covers unbound specs from create-table and commit requests, whose source
+	// IDs are client ordinal placeholders starting at 0, remapped by name at
+	// bind time.
 	for _, sourceID := range p.SourceIDs {
-		_, isVoid := p.Transform.(VoidTransform)
-		if sourceID <= 0 && (!isVoid || hasSourceID || hasSourceIDs) {
-			return fmt.Errorf("%w: partition source ID must be positive: %d", ErrInvalidPartitionSpec, sourceID)
+		if sourceID < 0 {

Review Comment:
   I'd keep this check strict and move the ordinal-0 allowance to a request-specific path. `PartitionField.UnmarshalJSON` is the same function `ParseMetadataBytes` uses to parse persisted v1/v2/v3 metadata, so relaxing `<= 0` to `< 0` here lets an already-bound persisted spec carry source-id 0 with no error, and `sourceIdToFields` ends up keyed on 0. It also quietly drops the old `VoidTransform` carve-out, so `{"source-id":0,"transform":"void"}` used to be rejected and now isn't.
   
   The create-table path doesn't need the parser relaxed, `reassignIDs` already remaps ordinals by name via `previousMapFn(f.SourceID())`, so I'd leave this rejecting source-id `<= 0` and add an unbound decode path for requests. Same shape on the sort side in `validateSortSourceID`. wdyt?



##########
table/metadata_internal_test.go:
##########
@@ -711,6 +711,83 @@ func TestRejectStructurallyInvalidHistoricalPartitionSpec(t *testing.T) {
 	assert.ErrorContains(t, err, "spec ID must be non-negative")
 }
 
+// A create-table request carries unbound placeholder IDs rather than final
+// field IDs. Spark numbers the root struct's fields by ordinal, so the first
+// column is field-id 0 and partitioning or sorting by it yields source-id 0.
+// NewMetadata must accept that and remap every source ID by name, matching
+// Java's TableMetadata.newTableMetadata.
+func TestNewMetadataFromOrdinalNumberedRequest(t *testing.T) {
+	const requestSchema = `{
+		"type": "struct", "schema-id": 0,
+		"fields": [
+			{"id": 0, "name": "my_ints", "required": false, "type": "int"},
+			{"id": 1, "name": "my_floats", "required": false, "type": "double"},
+			{"id": 2, "name": "strings", "required": false, "type": "string"}
+		]
+	}`
+
+	for _, tt := range []struct {
+		name         string
+		spec         string
+		wantSourceID int
+		wantName     string
+	}{
+		{
+			name:         "identity on first column",
+			spec:         `{"spec-id":0,"fields":[{"name":"my_ints","transform":"identity","source-id":0,"field-id":1000}]}`,
+			wantSourceID: 1,
+			wantName:     "my_ints",
+		},
+		{
+			name:         "bucket on first column",
+			spec:         `{"spec-id":0,"fields":[{"name":"my_ints_bucket","transform":"bucket[16]","source-id":0,"field-id":1000}]}`,
+			wantSourceID: 1,
+			wantName:     "my_ints_bucket",
+		},
+		{
+			name:         "identity on last column",
+			spec:         `{"spec-id":0,"fields":[{"name":"strings","transform":"identity","source-id":2,"field-id":1000}]}`,
+			wantSourceID: 3,
+			wantName:     "strings",
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			var sc iceberg.Schema
+			require.NoError(t, json.Unmarshal([]byte(requestSchema), &sc))
+
+			var spec iceberg.PartitionSpec
+			require.NoError(t, json.Unmarshal([]byte(tt.spec), &spec))
+
+			var order SortOrder
+			require.NoError(t, json.Unmarshal([]byte(
+				`{"order-id":1,"fields":[{"transform":"identity","source-id":0,"direction":"asc","null-order":"nulls-first"}]}`,
+			), &order))
+
+			meta, err := NewMetadata(&sc, &spec, order, "s3://bucket/tbl", nil)
+			require.NoError(t, err)
+
+			gotSpec := meta.PartitionSpec()
+			require.Equal(t, 1, gotSpec.NumFields())
+
+			assert.Equal(t, tt.wantSourceID, gotSpec.Field(0).SourceID())
+			assert.Equal(t, tt.wantName, gotSpec.Field(0).Name)
+
+			// The sort order references the first column, so it remaps to 1.
+			for _, field := range meta.SortOrder().Fields() {

Review Comment:
   This loop can pass without asserting anything. If `meta.SortOrder()` comes back as `UnsortedSortOrder` (0 fields) because the sort remap silently failed, the range body never runs and the assertion is vacuous, so the exact regression this test is meant to catch would slip through green.
   
   I'd add `assert.Equal(t, 1, meta.SortOrder().Len())` right before the loop, mirroring the `require.Equal(t, 1, gotSpec.NumFields())` guard you already have on the partition side.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3750086974


##########
partitions.go:
##########
@@ -162,10 +162,13 @@ func (p *PartitionField) UnmarshalJSON(b []byte) error {
 	} else {
 		p.SourceIDs = []int{aux.SourceID}
 	}
+	// Positivity only holds for specs already bound to a schema. Decoding also
+	// covers unbound specs from create-table and commit requests, whose source
+	// IDs are client ordinal placeholders starting at 0, remapped by name at
+	// bind time.
 	for _, sourceID := range p.SourceIDs {
-		_, isVoid := p.Transform.(VoidTransform)
-		if sourceID <= 0 && (!isVoid || hasSourceID || hasSourceIDs) {
-			return fmt.Errorf("%w: partition source ID must be positive: %d", ErrInvalidPartitionSpec, sourceID)
+		if sourceID < 0 {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#issuecomment-5241351783

   > Duplicate temporary IDs remain a separate issue. Multiple zeros can make the name lookup used by table/metadata.go:2814-2841 nondeterministic. Suggested fix: reject duplicate temporary IDs before reassignment. Please add persisted v1/v2/v3 rejection tests covering default and historical specs/orders, shuffled IDs proving exact matching, duplicate-zero rejection, and round-trip checks that write paths emit only positive IDs.
   
   @zeroshade I captured what Spark actually posts, by pointing spark.sql.catalog.probe.uri at a stub REST catalog that logs the create-table body. For `CREATE TABLE t (ints INT, floats DOUBLE, strings STRING) USING iceberg PARTITIONED BY (ints, bucket(16, ints))` the request carries `"partition-spec": {"spec-id":0,"fields":[{"name":"i
     nts","transform":"identity","source-id":0,"field-id":1000},{"name":"ints_bucket","transform":"bucket[16]","source-id":0,"field-id":1001}]}`, so duplicate placeholders are the normal wire form for partitioning one column two ways and rejecting them would reject that statement. They also resolve deterministically: `reassignIDs` looks each field up with `sc.FindColumnName(f.SourceID())` (table/metadata.go:2815,2828) over a `map[int]string` (schema.go:364-369), so both fields resolve to ints. Duplicates that are redundant never reach us, Spark refuses `PARTITIONED BY (years(ts), months(ts))` client-side with `IllegalArgumentException: Cannot add redundant partition: 1000: ts_year: year(0) conflicts with 1001: ts_month: month(0)` at `PartitionSpec$Builder.checkForRedundantPartitions(PartitionSpec.java:424)`, and identity twice on one placeholder is rejected by our own `validatePartitionFields`(partitions.go:564). The test cases now use these captured bodies verbatim.
   
   Separately worth its own issue: our redundancy key is the exact transform string with no dedupName equivalent, so `year(ts) + month(ts)` on one column would be accepted if a non-Spark client sent it. If you'd still rather reject duplicate placeholders outright, I'll switch it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#issuecomment-5241361209

   also updated PR description


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3750109438


##########
table/metadata_internal_test.go:
##########
@@ -711,6 +711,83 @@ func TestRejectStructurallyInvalidHistoricalPartitionSpec(t *testing.T) {
 	assert.ErrorContains(t, err, "spec ID must be non-negative")
 }
 
+// A create-table request carries unbound placeholder IDs rather than final
+// field IDs. Spark numbers the root struct's fields by ordinal, so the first
+// column is field-id 0 and partitioning or sorting by it yields source-id 0.
+// NewMetadata must accept that and remap every source ID by name, matching
+// Java's TableMetadata.newTableMetadata.
+func TestNewMetadataFromOrdinalNumberedRequest(t *testing.T) {

Review Comment:
   Added `TestRejectZeroSourceIDInPersistedMetadata` covering v1/v2/v3 across default and historical specs and orders, plus `TestRejectZeroSourceIDInV1LegacyPartitionSpec` for the bare partition-spec field list, and the create-table test now re-parses the written metadata with the strict parser as the write-path check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "fallintoplace (via GitHub)" <gi...@apache.org>.
fallintoplace commented on PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#issuecomment-5197904079

   Thanks for catching this. I think your solution is correct. I checked the Java implementation and Spark’s schema conversion, and they confirm that source ID 0 is valid as an ordinal placeholder during table creation before the IDs are reassigned.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3732111105


##########
table/sorting.go:
##########
@@ -191,9 +191,11 @@ func (s *SortField) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
+// Zero is valid: unbound sort orders carry client ordinal placeholder IDs that
+// start at 0. CheckCompatibility rejects IDs that don't resolve against a schema.
 func validateSortSourceID(id int) error {
-	if id <= 0 {
-		return fmt.Errorf("source ID must be positive: %d", id)
+	if id < 0 {

Review Comment:
   This global relaxation also admits zero into historical sort orders. Only the default order is compatibility-checked during metadata validation, so a historical order with source ID zero can persist. **Suggested fix:** limit zero to the unbound request path and structurally validate every persisted sort order with positive source IDs.



##########
partitions.go:
##########
@@ -162,10 +162,13 @@ func (p *PartitionField) UnmarshalJSON(b []byte) error {
 	} else {
 		p.SourceIDs = []int{aux.SourceID}
 	}
+	// Positivity only holds for specs already bound to a schema. Decoding also
+	// covers unbound specs from create-table and commit requests, whose source
+	// IDs are client ordinal placeholders starting at 0, remapped by name at
+	// bind time.
 	for _, sourceID := range p.SourceIDs {
-		_, isVoid := p.Transform.(VoidTransform)
-		if sourceID <= 0 && (!isVoid || hasSourceID || hasSourceIDs) {
-			return fmt.Errorf("%w: partition source ID must be positive: %d", ErrInvalidPartitionSpec, sourceID)
+		if sourceID < 0 {

Review Comment:
   This decoder is also used for persisted default and historical partition specs, so changing the global invariant admits source ID zero into stored metadata; `checkPartitionSpecs` does not bind or validate those source IDs. **Suggested fix:** accept temporary zero only in the unbound request path, then require positive IDs for all bound/persisted specs while retaining the existing source-less void tombstone exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3750089271


##########
table/sorting.go:
##########
@@ -191,9 +191,11 @@ func (s *SortField) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
+// Zero is valid: unbound sort orders carry client ordinal placeholder IDs that
+// start at 0. CheckCompatibility rejects IDs that don't resolve against a schema.
 func validateSortSourceID(id int) error {
-	if id <= 0 {
-		return fmt.Errorf("source ID must be positive: %d", id)
+	if id < 0 {

Review Comment:
   done, made strict again for bound orders



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3750098513


##########
partitions.go:
##########
@@ -162,10 +162,13 @@ func (p *PartitionField) UnmarshalJSON(b []byte) error {
 	} else {
 		p.SourceIDs = []int{aux.SourceID}
 	}
+	// Positivity only holds for specs already bound to a schema. Decoding also
+	// covers unbound specs from create-table and commit requests, whose source
+	// IDs are client ordinal placeholders starting at 0, remapped by name at
+	// bind time.
 	for _, sourceID := range p.SourceIDs {
-		_, isVoid := p.Transform.(VoidTransform)
-		if sourceID <= 0 && (!isVoid || hasSourceID || hasSourceIDs) {
-			return fmt.Errorf("%w: partition source ID must be positive: %d", ErrInvalidPartitionSpec, sourceID)
+		if sourceID < 0 {

Review Comment:
   done, the parser stays strict and the ordinal allowance lives in a request-specific unbound type, and the `{"source-id":0,"transform":"void"}` rejection is back too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#discussion_r3753493691


##########
table/sorting.go:
##########
@@ -195,21 +208,26 @@ func (s *SortField) UnmarshalJSON(b []byte) error {
 	return nil
 }
 
-func validateSortSourceID(id int) error {
-	if id <= 0 {
+// Source IDs are schema field IDs, and therefore positive, once an order is
+// bound to a schema. Unbound orders carry client placeholders that start at zero.
+func validateSortSourceID(id int, binding orderBinding) error {
+	if binding == boundOrder && id <= 0 {
 		return fmt.Errorf("source ID must be positive: %d", id)
 	}
+	if id < 0 {
+		return fmt.Errorf("source ID must be non-negative: %d", id)

Review Comment:
   Small consistency thing while we're here: the new non-negative branch returns a bare `fmt.Errorf` without wrapping `ErrInvalidSortSourceID`, whereas `validatePartitionSourceID` wraps its sentinel directly in both branches. It works today because `SortField.unmarshal` double-wraps at the call site, but a future caller of `validateSortSourceID` would silently lose the sentinel.
   
   Same theme one function down at `CheckCompatibility` (sorting.go:437), which interpolates the inner error with `%v` instead of `%w`. I'd wrap the sentinel here and switch that `%v` to `%w` so `errors.Is` keeps working through both paths.



##########
table/metadata_internal_test.go:
##########
@@ -743,6 +743,240 @@ func TestRejectStructurallyInvalidHistoricalPartitionSpec(t *testing.T) {
 	assert.ErrorContains(t, err, "spec ID must be non-negative")
 }
 
+// A create-table request carries unbound placeholder IDs rather than final
+// field IDs. Spark numbers the root struct's fields by ordinal, so the first
+// column is field-id 0 and partitioning by it yields source-id 0. NewMetadata
+// must accept an unbound spec and order and remap every source ID by name,
+// matching Java's TableMetadata.newTableMetadata.
+//
+// The schema and specs below are the bodies Spark 3.5 with
+// iceberg-spark-runtime posts to /v1/namespaces/{ns}/tables for
+// CREATE TABLE t (ints INT, floats DOUBLE, strings STRING) USING iceberg
+// PARTITIONED BY (...).
+func TestNewMetadataFromOrdinalNumberedRequest(t *testing.T) {

Review Comment:
   One coverage gap worth closing here. Every sub-case remaps a placeholder that happens to line up with a schema field ID (source-id 0 against a schema whose IDs start at 0), so a name-based remap and an ID-based remap would give the same answer. That leaves the interesting cases untested: a placeholder with no matching column in the new schema (say source-id 5 against a 3-column schema), and a schema whose field IDs aren't the ordinals (5,7,9) paired with ordinal source IDs.
   
   I'd add a sub-case for the out-of-range placeholder asserting it errors rather than silently producing invalid metadata. That also documents whether the remap is intentionally constrained to the Spark ordinal wire format.



##########
partitions.go:
##########
@@ -190,6 +206,24 @@ type PartitionSpec struct {
 	sourceIdToFields map[int][]PartitionField
 }
 
+// UnboundPartitionSpec decodes a partition spec that a client sent in a
+// create-table request, before it has been bound to a schema. Such a spec
+// carries the client's placeholder source IDs rather than schema field IDs, and
+// those placeholders start at zero: Spark numbers the columns of a new table
+// from zero, so partitioning by the first column arrives as source-id 0.
+// Binding the embedded spec to a schema resolves the placeholders to field IDs.
+//
+// Use PartitionSpec for specs read from table metadata, where source IDs are
+// bound field IDs and must be positive. Catalog implementations that serve the
+// REST create-table request should decode its partition spec into this type.
+type UnboundPartitionSpec struct {

Review Comment:
   I think there's a subtle trap in exposing the embedded `PartitionSpec` here. `BindToSchema` resolves each field through `AddPartitionFieldBySourceID`, which looks the source up by field ID. On an unbound spec the source IDs are the client's zero-based placeholders, so `unboundSpec.BindToSchema(schema, ...)` hits `FindFieldByID(0)`, matches nothing, and errors out. The only correct path today is `NewMetadata` into `reassignIDs`, which resolves by name.
   
   That's easy to trip over since `BindToSchema` sits right there on the public surface. I'd either override it on `UnboundPartitionSpec` to remap by name (what Java's `UnboundPartitionSpec.bind` does), or add a doc line saying binding goes through `NewMetadata` and `BindToSchema` isn't valid on the unbound type. wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665#issuecomment-5315743981

   @laskoviymishka, dug a bit in iceberg-rust, they do all client-side things unbound, `TableCreation.partition_spec` is
   `Option<UnboundPartitionSpec>` (`crates/iceberg/src/catalog/mod.rs:330`), so is the wire struct `CreateTableRequest.partition_spec` (`crates/catalog/rest/src/types.rs:258`), and the REST client forwards one straight into the other (`crates/catalog/rest/src/catalog.rs:755`). The reasoning carries over: a spec passed to `CreateTable` references the caller's schema, whose IDs get reassigned, so it plays the same placeholder role as a decoded Spark request and both directions share one JSON shape, so one type should serve both.
   
   If we do it here, we need to re-type `catalog.WithPartitionSpec` which is the shared `CreateTableOpt` surface for REST, Glue, etc.  And would also give `UnboundPartitionSpec` a construction path. Rust has `into_unbound()` (`partition.rs:111`), `From<PartitionSpec>` and a builder, we only got `UnmarshalJSON`. That'd be a breaking change to a public API and should probably be discussed in separate and not get folded into this decode-side bug-fix.
   
   For now, I've left a note in `catalog/rest/rest.go:225` pointing at the unbound types & #1664.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(partitions,sort): accept ordinal source IDs when decoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #1665:
URL: https://github.com/apache/iceberg-go/pull/1665


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org