You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fallintoplace (via GitHub)" <gi...@apache.org> on 2026/08/04 21:42:33 UTC

[PR] fix(table): replace metadata state during JSON decoding [iceberg-go]

fallintoplace opened a new pull request, #1648:
URL: https://github.com/apache/iceberg-go/pull/1648

   ## Summary
   
   Make table metadata JSON decoding replace the receiver only after the complete document validates.
   
   ## Why
   
   The v1, v2, and v3 decoders currently unmarshal through aliases of the existing receiver. Reusing metadata can retain omitted optional fields, while invalid input can leave a partially updated object behind.
   
   ## Testing
   
   - `go test ./table`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): replace metadata state during JSON decoding [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #1648:
URL: https://github.com/apache/iceberg-go/pull/1648#issuecomment-5219738561

   Re-reviewed after `95a0d56` and `1efe8e2`. These pick up six of the seven notes from my last pass, and several of them go further than I asked — seeding the fixture with the previously-absent optional fields is a better fix for the vacuous-v1 problem than the guard assertions I suggested, and overriding `last-partition-id` to 1001 rather than deleting it tests replacement instead of just absence. Thanks for working through all of them.
   
   For the record, what landed:
   
   - `ParseMetadataBytes` now builds `&metadataV1{}` / `&metadataV2{}` / `&metadataV3{}` (`table/metadata.go:1514-1518`).
   - The three dead `aux.LastColumnId = -1` assignments are gone, with the explanation moved to `initCommonMetadataForDeserialization` (`table/metadata.go:1662`) where the value originates.
   - The delete-list gained `statistics`, `partition-statistics`, `encryption-keys`, and `last-sequence-number`.
   - Failure-path coverage is table-driven across all three versions.
   - The failure-path assertion is now a `json.Marshal` before/after byte comparison.
   
   Two things I found this round, both cheap, and then one wording item that is still outstanding.
   
   **1. `TestMetadataUnmarshalPreservesStateOnError/v1` does not currently guard anything.**
   
   I checked this by reverting all three `UnmarshalJSON`s to decode into the receiver via `aux := (*Alias)(m)`. The v2 and v3 subtests fail, as they should. The v1 subtest passes.
   
   The reason is that the invalid v1 document differs from the valid one only inside `"schema"`, and the error it raises — `invalid schema: multiple fields for id 1: x and y` — comes out of `json.Unmarshal` itself. Every field the decoder writes before reaching that point is byte-identical to what the receiver already holds, so `before == after` whether or not the fix is present.
   
   This matters because v1 is the version with the largest partial-mutation surface: `metadataV1.preValidate` (`table/metadata.go:2432-2464`) mutates `SchemaList`, `Specs`, `DefaultSpecID`, `LastPartitionID`, and `SortOrderList` before validation runs.
   
   The fix is to perturb a top-level scalar so a partial write is observable. I verified two candidates — both fail with the fix reverted and pass with it in place:
   
   - `"current-snapshot-id": -1` → `999`, which fails with `invalid metadata: snapshot ref main with ID 999 does not exist in snapshot list`. This is the direct parallel to what the v2 and v3 subtests already do.
   - `"last-column-id": 3` → `-5`, which fails with `invalid metadata: missing last-column-id`.
   
   Either works; the first keeps the three subtests symmetric.
   
   **2. The new `assert.Equal(t, int64(-1), metadata.LastSeqNum)` pins a pre-existing bug as expected behavior.**
   
   I raised this leak last time as non-blocking, and it is genuinely pre-existing rather than anything this PR introduces — I confirmed that on `main`. But it is now asserted, which changes its status: a future fix to the sentinel normalization will break this line and read as a regression.
   
   To be concrete about what is being pinned: for a v2 document that omits `last-sequence-number` and has no snapshots, `checkLastSequenceNumber` (`table/metadata.go:2709`) only iterates the snapshot list, so with zero snapshots nothing checks it. On `main` today, `ParseMetadataBytes` on such a document returns successfully, `LastSequenceNumber()` reports `-1`, and re-serializing emits `"last-sequence-number": -1` — an invalid v2 document, through the public API.
   
   A short comment on the assertion noting that `-1` is the un-normalized sentinel and not the desired end state would be enough. Happy to file the underlying issue separately.
   
   **3. Still outstanding from last time: the description.**
   
   `ccc5e4e` has no commit body, so the wording lives only in the PR description, which still reads "Existing version-specific defaults and sentinels are preserved". They are now applied *unconditionally*, which makes direct `UnmarshalJSON` strictly more rejecting — `default-spec-id` becomes required for v2 and `next-row-id` for v3, matching what `ParseMetadataBytes` already enforced. That is the right behavior and unobservable externally since `metadataV{1,2,3}` are unexported, but it is a tightening rather than a preservation, and the description is what carries into the record on squash.
   
   Nothing here touches the production change, which I verified last pass and which is unchanged. On my side: `go test ./table/... ./ -count=1` green, `gofmt -l table/` and `go vet ./table/...` clean, all 15 CI checks passing.
   
   Happy to approve once the v1 fixture is swapped and the description is updated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): replace metadata state during JSON decoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #1648:
URL: https://github.com/apache/iceberg-go/pull/1648


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): replace metadata state during JSON decoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1648:
URL: https://github.com/apache/iceberg-go/pull/1648#discussion_r3740298879


##########
table/metadata.go:
##########
@@ -1692,6 +1692,7 @@ type commonMetadata struct {
 
 func initCommonMetadataForDeserialization() commonMetadata {
 	return commonMetadata{
+		// Negative sentinels distinguish omitted required fields from explicit zero values.
 		LastColumnId:       -1,

Review Comment:
   This comment promises more than the validators actually deliver.
   
   `LastUpdatedMS` is set to -1 in the same initializer, but `validate()` checks `== 0`, so a document that omits `last-updated-ms` keeps the -1 sentinel and passes. The "distinguish omitted from explicit zero" invariant doesn't hold for it.
   
   Same shape on `LastSeqNum`, which is the -1 round-trip zeroshade flagged and which you now document in the reuse test. It's spec-required for v2/v3, so a `validate()` guard would close it properly, but I'd be fine scoping this comment to the fields that are really enforced now and leaving the guard as a follow-up. wdyt?



##########
table/metadata.go:
##########
@@ -2653,18 +2662,22 @@ func (m *metadataV3) Equals(other Metadata) bool {
 
 func (m *metadataV3) UnmarshalJSON(b []byte) error {
 	type Alias metadataV3
-	aux := (*Alias)(m)
-
-	// Set LastColumnId to -1 to indicate that it is not set as LastColumnId = 0 is a valid value for when no schema is present
-	aux.LastColumnId = -1
+	next := initMetadataV3Deser()
+	aux := (*Alias)(next)
 
 	if err := json.Unmarshal(b, aux); err != nil {
 		return err
 	}
 
-	m.preValidate()
+	next.preValidate()

Review Comment:
   v1 and v2 `UnmarshalJSON` both call `rejectFieldsBeyondVersion`, v3 doesn't (pre-existing, but this is the natural spot since the PR reworks this body). It's harmless today, but once v4 lands, a v3 object read via `json.Unmarshal` would silently accept v4-only fields where Java's parser rejects them.
   
   Adding the call now with an empty field list documents the pattern and makes the v4 addition mechanical. Non-blocking, up to you whether it's in scope here.



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,187 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			initialData := []byte(tt.data)
+			var initial map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal(initialData, &initial))
+			setJSON := func(key string, value any) {
+				encoded, err := json.Marshal(value)
+				require.NoError(t, err)
+				initial[key] = encoded
+			}
+			snapshotID := int64(1925)
+			if tt.name != "v1" {
+				snapshotID = 3055729675574597004
+			}
+			setJSON("properties", map[string]any{"seed": "value"})
+			setJSON("current-snapshot-id", snapshotID)
+			setJSON("snapshot-log", []any{map[string]any{"snapshot-id": snapshotID, "timestamp-ms": int64(1)}})
+			setJSON("metadata-log", []any{map[string]any{"metadata-file": "s3://bucket/old.json", "timestamp-ms": int64(1)}})
+			setJSON("refs", map[string]any{"main": map[string]any{"snapshot-id": snapshotID, "type": "branch"}})
+			setJSON("last-partition-id", int64(1000))
+			setJSON("statistics", []any{map[string]any{
+				"snapshot-id": snapshotID, "statistics-path": "s3://bucket/stats.puffin",
+				"file-size-in-bytes": int64(1), "file-footer-size-in-bytes": int64(1), "blob-metadata": []any{},
+			}})
+			setJSON("partition-statistics", []any{map[string]any{
+				"snapshot-id": snapshotID, "statistics-path": "s3://bucket/partition-stats.parquet", "file-size-in-bytes": int64(1),
+			}})
+			if tt.name == "v2" || tt.name == "v3" {
+				setJSON("last-sequence-number", int64(34))
+			}
+			if tt.name == "v3" {
+				setJSON("encryption-keys", []any{map[string]any{
+					"key-id": "key-1", "encrypted-key-metadata": "YWJj",
+				}})
+			}
+			var err error
+			initialData, err = json.Marshal(initial)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(initialData, tt.target))
+
+			common := metadataCommon(tt.target)
+			require.NotEmpty(t, common.Props)
+			require.NotEmpty(t, common.SnapshotList)
+			require.NotEmpty(t, common.SnapshotLog)
+			require.NotEmpty(t, common.MetadataLog)
+			require.NotEmpty(t, common.SnapshotRefs)
+			require.NotNil(t, common.CurrentSnapshotID)
+			require.NotNil(t, common.LastPartitionID)
+			require.NotEmpty(t, common.StatisticsList)
+			require.NotEmpty(t, common.PartitionStatsList)
+			switch metadata := tt.target.(type) {
+			case *metadataV2:
+				require.Equal(t, int64(34), metadata.LastSeqNum)
+			case *metadataV3:
+				require.Equal(t, int64(34), metadata.LastSeqNum)
+			}
+			if tt.name == "v3" {
+				require.NotEmpty(t, common.EncryptionKeyList)
+			}
+
+			var reduced map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal([]byte(tt.data), &reduced))
+			for _, key := range []string{
+				"properties", "current-snapshot-id", "snapshots", "snapshot-log", "metadata-log", "refs",
+				"statistics", "partition-statistics", "encryption-keys", "last-sequence-number",
+			} {
+				delete(reduced, key)
+			}
+			reduced["last-partition-id"] = json.RawMessage("1001")
+			reducedData, err := json.Marshal(reduced)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(reducedData, tt.target))
+
+			common = metadataCommon(tt.target)
+			assert.Empty(t, common.Props)
+			assert.Empty(t, common.SnapshotList)
+			assert.Empty(t, common.SnapshotLog)
+			assert.Empty(t, common.MetadataLog)
+			assert.Empty(t, common.SnapshotRefs)
+			assert.Nil(t, common.CurrentSnapshotID)
+			assert.Equal(t, 1001, *common.LastPartitionID)
+			assert.Empty(t, common.StatisticsList)
+			assert.Empty(t, common.PartitionStatsList)
+			assert.Empty(t, common.EncryptionKeyList)
+			// With no snapshots, an omitted last-sequence-number retains the
+			// legacy -1 sentinel. This documents current behavior, not the
+			// desired metadata normalization.
+			switch metadata := tt.target.(type) {
+			case *metadataV2:
+				assert.Equal(t, int64(-1), metadata.LastSeqNum)
+			case *metadataV3:
+				assert.Equal(t, int64(-1), metadata.LastSeqNum)
+			}
+		})
+	}
+}
+
+func metadataCommon(target any) *commonMetadata {
+	switch metadata := target.(type) {
+	case *metadataV1:
+		return &metadata.commonMetadata
+	case *metadataV2:
+		return &metadata.commonMetadata
+	case *metadataV3:
+		return &metadata.commonMetadata
+	default:
+		panic("unsupported metadata type")
+	}
+}
+
+func TestMetadataUnmarshalPreservesStateOnError(t *testing.T) {
+	tests := []struct {
+		name    string
+		data    string
+		invalid string
+		target  any
+	}{
+		{
+			name:    "v1",
+			data:    ExampleTableMetadataV1,
+			invalid: strings.Replace(ExampleTableMetadataV1, `"current-snapshot-id": -1`, `"current-snapshot-id": 999`, 1),
+			target:  &metadataV1{},
+		},
+		{
+			name:    "v2",
+			data:    ExampleTableMetadataV2,
+			invalid: strings.Replace(ExampleTableMetadataV2, `"current-schema-id": 1`, `"current-schema-id": 99`, 1),
+			target:  &metadataV2{},
+		},
+		{
+			name:    "v3",
+			data:    ExampleTableMetadataV3,
+			invalid: strings.Replace(ExampleTableMetadataV3, `"current-schema-id": 1`, `"current-schema-id": 99`, 1),
+			target:  &metadataV3{},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			require.NoError(t, json.Unmarshal([]byte(tt.data), tt.target))
+			before, err := json.Marshal(tt.target)
+			require.NoError(t, err)
+
+			require.Error(t, json.Unmarshal([]byte(tt.invalid), tt.target))
+
+			after, err := json.Marshal(tt.target)
+			require.NoError(t, err)
+			assert.Equal(t, before, after)
+		})
+	}
+}
+
+func TestMetadataV2UnmarshalRejectsMissingRequiredStateOnReuse(t *testing.T) {
+	var metadata metadataV2
+	require.NoError(t, json.Unmarshal([]byte(ExampleTableMetadataV2), &metadata))
+
+	var reduced map[string]json.RawMessage
+	require.NoError(t, json.Unmarshal([]byte(ExampleTableMetadataV2), &reduced))
+	delete(reduced, "default-spec-id")
+	reducedData, err := json.Marshal(reduced)
+	require.NoError(t, err)
+
+	require.Error(t, json.Unmarshal(reducedData, &metadata))
+	assert.Equal(t, 0, metadata.DefaultSpecID)

Review Comment:
   This assertion doesn't discriminate what we want it to. The preserved `DefaultSpecID` is 0, but 0 is also the Go zero value, so a hypothetical "zero-fill the receiver on failure" regression would pass this too.
   
   I'd seed the first decode with a non-zero `default-spec-id` and assert that value here. The `SnapshotList` length check below is the strong one, so keep that. (This test also overlaps `TestMetadataUnmarshalPreservesStateOnError` on the state-preservation angle, so you could narrow it to just the missing-field error if you'd rather not duplicate.)



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,187 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			initialData := []byte(tt.data)
+			var initial map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal(initialData, &initial))
+			setJSON := func(key string, value any) {
+				encoded, err := json.Marshal(value)
+				require.NoError(t, err)
+				initial[key] = encoded
+			}
+			snapshotID := int64(1925)
+			if tt.name != "v1" {
+				snapshotID = 3055729675574597004

Review Comment:
   This is the `current-snapshot-id` from `ExampleTableMetadataV2/V3` hardcoded as a literal. If the fixture's snapshot id ever changes, the test keeps using this value and fails as a confusing "snapshot not found" rather than pointing at the fixture.
   
   I'd pull it from the parsed fixture at setup or give it a named constant.



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,187 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			initialData := []byte(tt.data)
+			var initial map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal(initialData, &initial))
+			setJSON := func(key string, value any) {
+				encoded, err := json.Marshal(value)
+				require.NoError(t, err)
+				initial[key] = encoded
+			}
+			snapshotID := int64(1925)
+			if tt.name != "v1" {
+				snapshotID = 3055729675574597004
+			}
+			setJSON("properties", map[string]any{"seed": "value"})
+			setJSON("current-snapshot-id", snapshotID)
+			setJSON("snapshot-log", []any{map[string]any{"snapshot-id": snapshotID, "timestamp-ms": int64(1)}})
+			setJSON("metadata-log", []any{map[string]any{"metadata-file": "s3://bucket/old.json", "timestamp-ms": int64(1)}})
+			setJSON("refs", map[string]any{"main": map[string]any{"snapshot-id": snapshotID, "type": "branch"}})
+			setJSON("last-partition-id", int64(1000))
+			setJSON("statistics", []any{map[string]any{
+				"snapshot-id": snapshotID, "statistics-path": "s3://bucket/stats.puffin",
+				"file-size-in-bytes": int64(1), "file-footer-size-in-bytes": int64(1), "blob-metadata": []any{},
+			}})
+			setJSON("partition-statistics", []any{map[string]any{
+				"snapshot-id": snapshotID, "statistics-path": "s3://bucket/partition-stats.parquet", "file-size-in-bytes": int64(1),
+			}})
+			if tt.name == "v2" || tt.name == "v3" {
+				setJSON("last-sequence-number", int64(34))
+			}
+			if tt.name == "v3" {
+				setJSON("encryption-keys", []any{map[string]any{
+					"key-id": "key-1", "encrypted-key-metadata": "YWJj",
+				}})
+			}
+			var err error
+			initialData, err = json.Marshal(initial)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(initialData, tt.target))
+
+			common := metadataCommon(tt.target)
+			require.NotEmpty(t, common.Props)
+			require.NotEmpty(t, common.SnapshotList)
+			require.NotEmpty(t, common.SnapshotLog)
+			require.NotEmpty(t, common.MetadataLog)
+			require.NotEmpty(t, common.SnapshotRefs)
+			require.NotNil(t, common.CurrentSnapshotID)
+			require.NotNil(t, common.LastPartitionID)
+			require.NotEmpty(t, common.StatisticsList)
+			require.NotEmpty(t, common.PartitionStatsList)
+			switch metadata := tt.target.(type) {
+			case *metadataV2:
+				require.Equal(t, int64(34), metadata.LastSeqNum)
+			case *metadataV3:
+				require.Equal(t, int64(34), metadata.LastSeqNum)
+			}
+			if tt.name == "v3" {
+				require.NotEmpty(t, common.EncryptionKeyList)
+			}
+
+			var reduced map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal([]byte(tt.data), &reduced))
+			for _, key := range []string{
+				"properties", "current-snapshot-id", "snapshots", "snapshot-log", "metadata-log", "refs",
+				"statistics", "partition-statistics", "encryption-keys", "last-sequence-number",
+			} {
+				delete(reduced, key)
+			}
+			reduced["last-partition-id"] = json.RawMessage("1001")
+			reducedData, err := json.Marshal(reduced)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(reducedData, tt.target))
+
+			common = metadataCommon(tt.target)
+			assert.Empty(t, common.Props)
+			assert.Empty(t, common.SnapshotList)
+			assert.Empty(t, common.SnapshotLog)
+			assert.Empty(t, common.MetadataLog)
+			assert.Empty(t, common.SnapshotRefs)
+			assert.Nil(t, common.CurrentSnapshotID)
+			assert.Equal(t, 1001, *common.LastPartitionID)
+			assert.Empty(t, common.StatisticsList)
+			assert.Empty(t, common.PartitionStatsList)
+			assert.Empty(t, common.EncryptionKeyList)
+			// With no snapshots, an omitted last-sequence-number retains the
+			// legacy -1 sentinel. This documents current behavior, not the
+			// desired metadata normalization.
+			switch metadata := tt.target.(type) {
+			case *metadataV2:
+				assert.Equal(t, int64(-1), metadata.LastSeqNum)
+			case *metadataV3:
+				assert.Equal(t, int64(-1), metadata.LastSeqNum)
+			}
+		})
+	}
+}
+
+func metadataCommon(target any) *commonMetadata {
+	switch metadata := target.(type) {
+	case *metadataV1:
+		return &metadata.commonMetadata
+	case *metadataV2:
+		return &metadata.commonMetadata
+	case *metadataV3:
+		return &metadata.commonMetadata
+	default:
+		panic("unsupported metadata type")
+	}
+}
+
+func TestMetadataUnmarshalPreservesStateOnError(t *testing.T) {
+	tests := []struct {
+		name    string
+		data    string
+		invalid string
+		target  any
+	}{
+		{
+			name:    "v1",
+			data:    ExampleTableMetadataV1,
+			invalid: strings.Replace(ExampleTableMetadataV1, `"current-snapshot-id": -1`, `"current-snapshot-id": 999`, 1),
+			target:  &metadataV1{},
+		},
+		{
+			name:    "v2",
+			data:    ExampleTableMetadataV2,
+			invalid: strings.Replace(ExampleTableMetadataV2, `"current-schema-id": 1`, `"current-schema-id": 99`, 1),

Review Comment:
   All three invalid cases here are semantically invalid but syntactically valid, so they only exercise the `validate()` error path, which runs after `json.Unmarshal` succeeds. The earlier `json.Unmarshal` error path, the one that returns before `*m = *next`, isn't covered by any case.
   
   A fourth case per version with something like ``invalid: `not json {` `` would exercise it in a line each.



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,187 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			initialData := []byte(tt.data)
+			var initial map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal(initialData, &initial))
+			setJSON := func(key string, value any) {
+				encoded, err := json.Marshal(value)
+				require.NoError(t, err)
+				initial[key] = encoded
+			}
+			snapshotID := int64(1925)
+			if tt.name != "v1" {
+				snapshotID = 3055729675574597004
+			}
+			setJSON("properties", map[string]any{"seed": "value"})
+			setJSON("current-snapshot-id", snapshotID)
+			setJSON("snapshot-log", []any{map[string]any{"snapshot-id": snapshotID, "timestamp-ms": int64(1)}})
+			setJSON("metadata-log", []any{map[string]any{"metadata-file": "s3://bucket/old.json", "timestamp-ms": int64(1)}})
+			setJSON("refs", map[string]any{"main": map[string]any{"snapshot-id": snapshotID, "type": "branch"}})
+			setJSON("last-partition-id", int64(1000))
+			setJSON("statistics", []any{map[string]any{
+				"snapshot-id": snapshotID, "statistics-path": "s3://bucket/stats.puffin",
+				"file-size-in-bytes": int64(1), "file-footer-size-in-bytes": int64(1), "blob-metadata": []any{},
+			}})
+			setJSON("partition-statistics", []any{map[string]any{
+				"snapshot-id": snapshotID, "statistics-path": "s3://bucket/partition-stats.parquet", "file-size-in-bytes": int64(1),
+			}})
+			if tt.name == "v2" || tt.name == "v3" {
+				setJSON("last-sequence-number", int64(34))
+			}
+			if tt.name == "v3" {
+				setJSON("encryption-keys", []any{map[string]any{
+					"key-id": "key-1", "encrypted-key-metadata": "YWJj",
+				}})
+			}
+			var err error
+			initialData, err = json.Marshal(initial)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(initialData, tt.target))
+
+			common := metadataCommon(tt.target)
+			require.NotEmpty(t, common.Props)
+			require.NotEmpty(t, common.SnapshotList)
+			require.NotEmpty(t, common.SnapshotLog)
+			require.NotEmpty(t, common.MetadataLog)
+			require.NotEmpty(t, common.SnapshotRefs)
+			require.NotNil(t, common.CurrentSnapshotID)
+			require.NotNil(t, common.LastPartitionID)
+			require.NotEmpty(t, common.StatisticsList)
+			require.NotEmpty(t, common.PartitionStatsList)
+			switch metadata := tt.target.(type) {

Review Comment:
   I think both of these type switches will trip the exhaustive linter we run in CI. They cover `*metadataV2` and `*metadataV3` but there's no `case *metadataV1` and no `default`, and the second switch further down (line 427) has the same gap.
   
   Adding `case *metadataV1:` to both keeps CI green, and it means the test won't silently skip v1 if that type ever grows a `LastSeqNum`-like field.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): replace metadata state during JSON decoding [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1648:
URL: https://github.com/apache/iceberg-go/pull/1648#discussion_r3732167999


##########
table/metadata_internal_test.go:
##########
@@ -1402,6 +1478,7 @@ func TestMetadataV2Validation(t *testing.T) {
 		"current-schema-id": 0,
 		"last-partition-id": 1000,
 		"schemas": [{"type":"struct","schema-id":0,"fields":[]}],
+		"default-spec-id": 0,

Review Comment:
   This added line — and the matching one at `:1501` — is the proof that the description is inaccurate, and it is the main thing worth changing here.
   
   "Existing version-specific defaults and sentinels are preserved" reads as a no-op, but they are now applied unconditionally, so direct `UnmarshalJSON` is strictly more rejecting than before. `initMetadataV2Deser()` yields `DefaultSpecID = -1`, and `commonMetadata.preValidate` has no `-1 → max(specs)` fallback — only `metadataV1.preValidate` does, at `table/metadata.go:2444-2446`. That is exactly why these two fixtures had to gain `"default-spec-id": 0`. The same applies to v3 `next-row-id`: a zero-value receiver gave `0` and passed, whereas `-1` now reaches `checkNextRowID`'s "next-row-id is required for v3 tables" (`table/metadata.go:2705-2709`).
   
   This is the right behavior — it matches what `ParseMetadataBytes` already enforced, and `metadataV{1,2,3}` are unexported so no external caller can observe the change. Consider rewording the commit message to say so, e.g. "the deserialization sentinels are now applied on every decode path, so `default-spec-id` / `next-row-id` are required for v2/v3 documents decoded directly, matching `ParseMetadataBytes`."



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,80 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			require.NoError(t, json.Unmarshal([]byte(tt.data), tt.target))
+
+			var reduced map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal([]byte(tt.data), &reduced))
+			for _, key := range []string{
+				"properties", "current-snapshot-id", "snapshots", "snapshot-log", "metadata-log", "refs",

Review Comment:
   Non-blocking: this delete-list omits several `omitempty` `commonMetadata` fields in the same bug class — `statistics`, `partition-statistics`, `encryption-keys`, `last-partition-id`, and `last-sequence-number` would all have survived a reuse identically. `last-sequence-number` is free to add; the rest would need fixture extension first.



##########
table/metadata.go:
##########
@@ -2465,7 +2465,8 @@ func (m *metadataV1) preValidate() {
 
 func (m *metadataV1) UnmarshalJSON(b []byte) error {
 	type Alias metadataV1
-	aux := (*Alias)(m)
+	next := initMetadataV1Deser()

Review Comment:
   Worth adding to the description: this fixes two things beyond the stated one.
   
   First, `commonMetadata.preValidate()` injects a `main` entry into `SnapshotRefs` when `CurrentSnapshotID != nil` (`table/metadata.go:2161-2168`). On a reused receiver whose new document omitted `refs`, that injection mutated the **previous** document's map. `TestMetadataUnmarshalReplacesReceiverState`'s `assert.Nil(t, common.CurrentSnapshotID)` plus `assert.Empty(t, common.SnapshotRefs)` is exactly the right assertion for it.
   
   Second, v1's `if aux.CurrentSchemaID == -1 && aux.Schema != nil` branch (`table/metadata.go:2488-2495`) previously never fired on a zero-value receiver, because `CurrentSchemaID` was `0` rather than the `-1` sentinel. Direct `UnmarshalJSON` therefore rejected v1 documents that `ParseMetadataBytes` accepted. The two paths are now consistent.



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,80 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},

Review Comment:
   Non-blocking: the v1 subtest is close to vacuous. `ExampleTableMetadataV1` (`table/metadata_internal_test.go:181-199`) has `"properties": {}`, no `refs`, no `snapshot-log`, no `metadata-log`, and `"current-snapshot-id": -1` (normalized to nil). Five of the six `assert.Empty`/`assert.Nil` calls below therefore already hold *before* the second decode, and would pass with the fix reverted. Only `SnapshotList` is genuinely exercised for v1.
   
   Either extend the v1 fixture, or — cheaper and more durable — add guard assertions right after the first decode (`require.NotEmpty(t, common.Props)`, and so on) so no version's subtest can silently go vacuous as fixtures change.



##########
table/metadata.go:
##########
@@ -2465,7 +2465,8 @@ func (m *metadataV1) preValidate() {
 
 func (m *metadataV1) UnmarshalJSON(b []byte) error {
 	type Alias metadataV1
-	aux := (*Alias)(m)
+	next := initMetadataV1Deser()
+	aux := (*Alias)(next)
 
 	// Set LastColumnId to -1 to indicate that it is not set as LastColumnId = 0 is a valid value for when no schema is present
 	aux.LastColumnId = -1

Review Comment:
   Non-blocking: this assignment and its comment are now dead code. `next` comes from `initMetadataV1Deser()`, and `initCommonMetadataForDeserialization()` (`table/metadata.go:1660`) already sets `LastColumnId` to `-1`. Same at `:2554` and `:2637`.
   
   Leaving both makes it ambiguous which one is authoritative. Consider dropping the redundant assignments and keeping the explanatory comment at the `init...Deser` helper, where the value actually originates.



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,80 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			require.NoError(t, json.Unmarshal([]byte(tt.data), tt.target))
+
+			var reduced map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal([]byte(tt.data), &reduced))
+			for _, key := range []string{
+				"properties", "current-snapshot-id", "snapshots", "snapshot-log", "metadata-log", "refs",
+			} {
+				delete(reduced, key)
+			}
+			reducedData, err := json.Marshal(reduced)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(reducedData, tt.target))
+
+			var common *commonMetadata
+			switch metadata := tt.target.(type) {
+			case *metadataV1:
+				common = &metadata.commonMetadata
+			case *metadataV2:
+				common = &metadata.commonMetadata
+			case *metadataV3:
+				common = &metadata.commonMetadata
+			}
+			assert.Empty(t, common.Props)
+			assert.Empty(t, common.SnapshotList)
+			assert.Empty(t, common.SnapshotLog)
+			assert.Empty(t, common.MetadataLog)
+			assert.Empty(t, common.SnapshotRefs)
+			assert.Nil(t, common.CurrentSnapshotID)
+		})
+	}
+}
+
+func TestMetadataV2UnmarshalPreservesStateOnError(t *testing.T) {

Review Comment:
   Non-blocking: failure-path coverage exists only for v2, yet all three `UnmarshalJSON`s changed.
   
   v1 needs it most — `metadataV1.preValidate` (`table/metadata.go:2432-2464`) mutates `SchemaList`, `Specs`, `DefaultSpecID`, `LastPartitionID`, and `SortOrderList` before validation runs, by far the largest partial-mutation surface of the three. v3 likewise fails inside `checkNextRowID` after `preValidate` has already run.



##########
table/metadata_internal_test.go:
##########
@@ -324,6 +324,80 @@ func TestMetadataV3Parsing(t *testing.T) {
 	assert.Equal(t, int64(2000), *secondSnapshot.FirstRowID)
 }
 
+func TestMetadataUnmarshalReplacesReceiverState(t *testing.T) {
+	tests := []struct {
+		name   string
+		data   string
+		target any
+	}{
+		{name: "v1", data: ExampleTableMetadataV1, target: &metadataV1{}},
+		{name: "v2", data: ExampleTableMetadataV2, target: &metadataV2{}},
+		{name: "v3", data: ExampleTableMetadataV3, target: &metadataV3{}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			require.NoError(t, json.Unmarshal([]byte(tt.data), tt.target))
+
+			var reduced map[string]json.RawMessage
+			require.NoError(t, json.Unmarshal([]byte(tt.data), &reduced))
+			for _, key := range []string{
+				"properties", "current-snapshot-id", "snapshots", "snapshot-log", "metadata-log", "refs",
+			} {
+				delete(reduced, key)
+			}
+			reducedData, err := json.Marshal(reduced)
+			require.NoError(t, err)
+
+			require.NoError(t, json.Unmarshal(reducedData, tt.target))
+
+			var common *commonMetadata
+			switch metadata := tt.target.(type) {
+			case *metadataV1:
+				common = &metadata.commonMetadata
+			case *metadataV2:
+				common = &metadata.commonMetadata
+			case *metadataV3:
+				common = &metadata.commonMetadata
+			}
+			assert.Empty(t, common.Props)
+			assert.Empty(t, common.SnapshotList)
+			assert.Empty(t, common.SnapshotLog)
+			assert.Empty(t, common.MetadataLog)
+			assert.Empty(t, common.SnapshotRefs)
+			assert.Nil(t, common.CurrentSnapshotID)
+		})
+	}
+}
+
+func TestMetadataV2UnmarshalPreservesStateOnError(t *testing.T) {
+	var metadata metadataV2
+	require.NoError(t, json.Unmarshal([]byte(ExampleTableMetadataV2), &metadata))
+
+	invalid := strings.Replace(ExampleTableMetadataV2, `"current-schema-id": 1`, `"current-schema-id": 99`, 1)
+	require.Error(t, json.Unmarshal([]byte(invalid), &metadata))
+
+	assert.Equal(t, 1, metadata.CurrentSchemaID)

Review Comment:
   Non-blocking, and the same suggestion made on #1646: rather than three hand-picked fields, snapshot `json.Marshal(&metadata)` before the failing decode and assert the bytes are unchanged afterwards. That covers every serialized field at once and would catch a partial regression that these three assertions would miss.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org