You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "anvsk (via GitHub)" <gi...@apache.org> on 2025/08/20 08:50:59 UTC
[I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
anvsk opened a new issue, #538:
URL: https://github.com/apache/iceberg-go/issues/538
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
## Environment
* **iceberg-go**: v0.3.1
* **Catalog**: REST catalog (standard Iceberg REST API)
* **Go**: 1.21+
* **OS**: Linux/Windows (reproducible across)
## Summary
In v0.3.0, `table.NewAddSchemaUpdate(*Schema, lastColumnID, initial)` allowed callers to ensure the table’s `last-assigned-field-id` stayed **monotonic** even when deleting the column that previously had the highest field ID.
In **v0.3.1**, the API changed to `table.NewAddSchemaUpdate(*Schema)` (no `lastColumnID`). When we **only delete the highest-ID column(s)** and **add no new columns**, the library appears to derive `last-assigned-field-id` from the **new schema’s max field id**, which **decreases**. The REST catalog then rejects the commit with:
```
invalid_metadata: The specified metadata is not valid
```
Deleting a first/middle column works; deleting the **tail (max-ID) column** fails.
> Note: This repro excludes partition/sort references (i.e., we are not deleting a column referenced by the default spec or sort order).
## Steps to Reproduce
1. Start with a table whose current schema has fields, e.g.:
* `a` (id=1), `b` (id=2), `c` (id=3).
No partition/sort references to `c`.
2. Build a **new schema** that **removes `c`** and keeps `a`/`b` with the **same field IDs** (we don’t touch IDs).
3. Submit **two updates in one commit**:
* `AddSchema` using `table.NewAddSchemaUpdate(newSchema)`
* `SetCurrentSchema` using `table.NewSetCurrentSchemaUpdate(newSchemaID)`
(We obtain `newSchemaID` by running `b := table.MetadataBuilderFromBase(meta); id, _ := b.AddSchema(newSchema)`.)
4. Include concurrency requirements (optional but recommended):
* `AssertTableUUID(meta.UUID())`
* `AssertLastAssignedFieldID(oldLastID)` where `oldLastID` is **3** in this example.
5. `CommitTable(...)` → **fails** with `invalid_metadata`.
Minimal code sketch (v0.3.1 style):
```go
meta := tbl.Metadata()
oldLast := highestID(tbl.Schema()) // returns 3 in the example
// Build new schema that keeps a(id=1), b(id=2) only (delete c(id=3))
newSchema := buildSchemaKeepAB(tbl.Schema()) // preserves existing IDs
// Precompute new schema-id
b := table.MetadataBuilderFromBase(meta)
newSchemaID, err := b.AddSchema(newSchema)
if err != nil { panic(err) }
// Prepare updates (v0.3.1 API)
add := table.NewAddSchemaUpdate(newSchema) // no lastColumnID parameter anymore
set := table.NewSetCurrentSchemaUpdate(newSchemaID)
reqs := []table.Requirement{
table.AssertTableUUID(meta.UUID()),
table.AssertLastAssignedFieldID(int(oldLast)), // oldLast == 3
}
_, _, err = cat.CommitTable(ctx, tbl, reqs, []table.Update{add, set})
// => invalid_metadata when only deleting tail/highest-ID columns
```
## Expected Behavior
* Deleting columns (including the highest-ID column) should be allowed as long as:
* We do not change existing field IDs of the kept columns.
* `last-assigned-field-id` **does not decrease** (i.e., remains the previous value).
* In v0.3.0, passing `lastColumnID=oldLast` ensured monotonicity and commits succeeded.
## Actual Behavior
* With v0.3.1, `NewAddSchemaUpdate` cannot accept `lastColumnID`.
* When we only delete the max-ID column and add no new columns, the commit is rejected with `invalid_metadata`—apparently because the derived `last-assigned-field-id` regresses to the new schema’s max ID.
## Analysis
* Iceberg requires `last-assigned-field-id` to be **monotonic** (never decreases).
* In the “delete-tail-columns only” scenario, the **current** `last-assigned-field-id` is the old max (e.g., 3). The **new schema’s** max becomes smaller (e.g., 2).
If the client or server infers the counter from the new schema’s max, it violates monotonicity → `invalid_metadata`.
## Workarounds
* **Add a sentinel (dummy) column** in the same update with ID = `oldLast + 1` (e.g., `__compat_padding_...`), nullable, never used. This keeps the new schema’s max ≥ old max.
Or, more practically, add a real new column in the same change so max ID increases.
* (Less ideal) Maintain a fork that restores the older API (`NewAddSchemaUpdate(schema, lastColumnID, initial)`) or custom-craft the REST payload to set `last-column-id = oldLast`.
* Of course still ensure you’re not deleting a field referenced by partition spec or sort order (not the case in this repro).
## Proposal
* **API / behavior options:**
1. Re-introduce a way to set **`lastColumnID`** (or an equivalent parameter) on `AddSchema` in the Go client; or
2. Have the client compute `last-assigned-field-id` as `max(oldLastID, max(newSchema.FieldIDs))` so it never regresses; or
3. Provide a dedicated update or requirement to explicitly set/preserve `last-assigned-field-id` without requiring a dummy column.
* **Docs**: Clarify in v0.3.1 migration notes that callers must ensure the counter doesn’t regress when deleting the highest-ID column, and suggest recommended patterns.
## Additional Context
* The same flow succeeds if we delete a middle/first column (the new schema’s max ID stays the same).
* The same flow succeeds if we add at least one new column (the new schema’s max ID increases).
Happy to provide a tiny repro program if needed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-4279290685
@alliasgher, this is not a bug in iceberg-go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "alliasgher (via GitHub)" <gi...@apache.org>.
alliasgher commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-4275928500
I'd like to work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3541505472
I'm not sure what s3tables' iceberg implementation is based on - if it is based on iceberg-java, then the reason for the error is likely this: https://github.com/apache/iceberg/issues/13850 with a proposed fix of: https://github.com/apache/iceberg/pull/14151
I'm not opposed to re-adding the parameter as optional until this is sorted out in iceberg-java, you could create a PR to this end. The proper solution is to fix this in iceberg-java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3213379926
Could you provide a complete reproducer for current main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3231746512
> The error looks like a go error, could you set a breakpoint in `doPost` and get the raw response, e.g. via:
>
> diff --git a/catalog/rest/rest.go b/catalog/rest/rest.go
> index 63c7389..7683757 100644
> --- a/catalog/rest/rest.go
> +++ b/catalog/rest/rest.go
> @@ -322,7 +322,8 @@ func doPost[Payload, Result any](ctx context.Context, baseURI *url.URL, path []s
> if rsp.StatusCode != http.StatusOK {
> return ret, handleNon200(rsp, override)
> }
> -
> + content, err := io.ReadAll(rsp.Body)
> + fmt.Println(rsp.StatusCode, string(content))
> if rsp.ContentLength == 0 {
> return
> }
<img width="2296" height="1112" alt="Image" src="https://github.com/user-attachments/assets/a0419838-4e97-44f1-8137-732d9e7f4624" />
change the last column ,this is success ,but when i delete the last column :

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3226823625
> s3tables is returning bad-request when iceberg-go is not sending `last-column-id`?
yes,return the error "invalid_metadata: The specified metadata is not valid"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3219460064
> Not sure I follow - does the server reject the commit? What catalog are you using?
rest catalog,the request payload without lastcolumnid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3210133642
Hi @anvsk, I think https://github.com/apache/iceberg-go/pull/539 fixes this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3212764482
> Hi [@anvsk](https://github.com/anvsk), I think [#539](https://github.com/apache/iceberg-go/pull/539) fixes this, could you verify?
i try and failed again
The new code seems to add a validation/guard only. When the input schema only deletes the tail column(s) (i.e., the column(s) with the highest field ID) and we don’t add any new columns, the client now fails that new check immediately and aborts. In other words, the last-assigned-field-id “regression” is detected, but there’s still no way for the client to preserve the previous lastAssignedFieldId for this commit, so the update remains impossible.
Either:
1. Re-introduce a way to pass or pin lastAssignedFieldId for AddSchema, or
2. Have the client compute it as max(oldLastId, max(newSchema.FieldIDs)) so it never regresses, or
3. Provide a dedicated update to explicitly set/preserve lastAssignedFieldId during schema evolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3219199039
> This test I hacked into `table/transaction_test.go` is also passing:
>
> func (s *SparkIntegrationTestSuite) TestAddSchemaRegression() {
> icebergSchema := iceberg.NewSchema(0,
> iceberg.NestedField{ID: 1, Name: "foo", Type: iceberg.PrimitiveTypes.Bool},
> iceberg.NestedField{ID: 2, Name: "bar", Type: iceberg.PrimitiveTypes.String},
> iceberg.NestedField{ID: 3, Name: "baz", Type: iceberg.PrimitiveTypes.Int32},
> )
>
> tbl, err := s.cat.CreateTable(s.ctx, catalog.ToIdentifier("default", "go_test_set_properties"), icebergSchema)
> s.Require().NoError(err)
> meta := tbl.Metadata()
> oldLast := tbl.Schema().HighestFieldID()
>
> icebergSchema2 := iceberg.NewSchema(0,
> iceberg.NestedField{ID: 1, Name: "foo", Type: iceberg.PrimitiveTypes.Bool},
> iceberg.NestedField{ID: 2, Name: "bar", Type: iceberg.PrimitiveTypes.String},
> )
>
> add := table.NewAddSchemaUpdate(icebergSchema2)
> set := table.NewSetCurrentSchemaUpdate(-1)
> reqs := []table.Requirement{
> table.AssertTableUUID(meta.TableUUID()),
> table.AssertLastAssignedFieldID(oldLast),
> }
>
> _, _, err = s.cat.CommitTable(s.ctx, tbl, reqs, []table.Update{add, set})
> s.Require().NoError(err)
> }
<img width="1341" height="974" alt="Image" src="https://github.com/user-attachments/assets/cb43b667-bf60-47c0-991c-f24972d99350" />
thank you ! i debug my code , but not go into your code func , i am using the catalog of rest , it just do a post request,so it lost the last column id set , and return "invalid_metadata",so i think table.addSchemaUpdate should add field lastColumnID
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "alliasgher (via GitHub)" <gi...@apache.org>.
alliasgher commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-4275946942
Opened #922 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3541453594
> That's strange, the catalog receives only a list of updates & requirements, not sure why it would return `The specified metadata is not valid` instead of rejecting the update, given that s3tables is probably used a bunch, we should be compatible to them. I guess we should re-introduce last-column-id as optional.
so,can you support last-column-id as optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "anvsk (via GitHub)" <gi...@apache.org>.
anvsk commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3220466281
> Which part of your setup is rejecting the request? The client-side rest catalog implementation of iceberg-go? The rest catalog server you are using?
>
> The last-column-id field as part of table update is deprecated and marked for removal: https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml
>
> <img alt="Image" width="562" height="95" src="https://private-user-images.githubusercontent.com/23434860/481551410-100c6402-6386-4516-9ae0-be742609daf0.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTYxMzA5OTgsIm5iZiI6MTc1NjEzMDY5OCwicGF0aCI6Ii8yMzQzNDg2MC80ODE1NTE0MTAtMTAwYzY0MDItNjM4Ni00NTE2LTlhZTAtYmU3NDI2MDlkYWYwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwODI1VDE0MDQ1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI2MWQwOTkwMWRjMDZlM2ZkNDg1ODU5MzY4NDUwNDA2OTNiODIxOTJlNzdkMjJkODhjNzBlYjNhOTZhNDkwNjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.dbq09sbC25K8AZRoY93TGk6qS6XsOhj-9cR5aZfAIY0">
i use the rest catalog server https://s3tables.ap-southeast-1.amazonaws.com/iceberg so ,this problem should be solved by the server side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3227640809
The error looks like a go error, could you set a breakpoint in `doPost` and get the raw response, e.g. via:
```patch
diff --git a/catalog/rest/rest.go b/catalog/rest/rest.go
index 63c7389..7683757 100644
--- a/catalog/rest/rest.go
+++ b/catalog/rest/rest.go
@@ -322,7 +322,8 @@ func doPost[Payload, Result any](ctx context.Context, baseURI *url.URL, path []s
if rsp.StatusCode != http.StatusOK {
return ret, handleNon200(rsp, override)
}
-
+ content, err := io.ReadAll(rsp.Body)
+ fmt.Println(rsp.StatusCode, string(content))
if rsp.ContentLength == 0 {
return
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3213516217
This test I hacked into `table/transaction_test.go` is also passing:
```go
func (s *SparkIntegrationTestSuite) TestAddSchemaRegression() {
icebergSchema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "foo", Type: iceberg.PrimitiveTypes.Bool},
iceberg.NestedField{ID: 2, Name: "bar", Type: iceberg.PrimitiveTypes.String},
iceberg.NestedField{ID: 3, Name: "baz", Type: iceberg.PrimitiveTypes.Int32},
)
tbl, err := s.cat.CreateTable(s.ctx, catalog.ToIdentifier("default", "go_test_set_properties"), icebergSchema)
s.Require().NoError(err)
meta := tbl.Metadata()
oldLast := tbl.Schema().HighestFieldID()
icebergSchema2 := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "foo", Type: iceberg.PrimitiveTypes.Bool},
iceberg.NestedField{ID: 2, Name: "bar", Type: iceberg.PrimitiveTypes.String},
)
add := table.NewAddSchemaUpdate(icebergSchema2)
set := table.NewSetCurrentSchemaUpdate(-1)
reqs := []table.Requirement{
table.AssertTableUUID(meta.TableUUID()),
table.AssertLastAssignedFieldID(oldLast),
}
_, _, err = s.cat.CommitTable(s.ctx, tbl, reqs, []table.Update{add, set})
s.Require().NoError(err)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3213376320
This test is passing @anvsk:
```go
func schema() iceberg.Schema {
return *iceberg.NewSchema(
0,
iceberg.NestedField{ID: 1, Name: "x", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "y", Type: iceberg.PrimitiveTypes.Int64, Required: true, Doc: "comment"},
iceberg.NestedField{ID: 3, Name: "z", Type: iceberg.PrimitiveTypes.Int64, Required: true},
)
}
func builderWithoutChanges(formatVersion int) MetadataBuilder {
tableSchema := schema()
partitionSpec := partitionSpec()
sortOrder := sortOrder()
builder, err := NewMetadataBuilder()
if err != nil {
panic(err)
}
_, err = builder.SetFormatVersion(formatVersion)
if err != nil {
panic(err)
}
_, err = builder.AddSortOrder(&sortOrder, true)
if err != nil {
panic(err)
}
_, err = builder.AddPartitionSpec(&partitionSpec, true)
if err != nil {
panic(err)
}
_, err = builder.AddSchema(&tableSchema)
if err != nil {
panic(err)
}
meta, err := builder.Build()
if err != nil {
panic(err)
}
builder, err = MetadataBuilderFromBase(meta)
if err != nil {
panic(err)
}
return *builder
}
func TestAddTwoSchemas(t *testing.T) {
builder := builderWithoutChanges(2)
schema2 := iceberg.NewSchema(
0,
iceberg.NestedField{ID: 1, Name: "x", Type: iceberg.PrimitiveTypes.Int64, Required: true},
iceberg.NestedField{ID: 2, Name: "y", Type: iceberg.PrimitiveTypes.Int64, Required: true, Doc: "comment"},
)
_, err := builder.AddSchema(schema2)
require.NoError(t, err)
require.Len(t, builder.schemaList, 2)
meta, err := builder.Build()
require.NoError(t, err)
require.NotNil(t, meta)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3223337295
s3tables is returning bad-request when iceberg-go is not sending `last-column-id`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3219307055
Not sure I follow - does the server reject the commit? What catalog are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3219473707
Which part of your setup is rejecting the request? The client-side rest catalog implementation of iceberg-go? The rest catalog server you are using?
The last-column-id field as part of table update is deprecated and marked for removal: https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml
<img width="562" height="95" alt="Image" src="https://github.com/user-attachments/assets/100c6402-6386-4516-9ae0-be742609daf0" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3234322782
That's strange, the catalog receives only a list of updates & requirements, not sure why it would return `The specified metadata is not valid` instead of rejecting the update, given that s3tables is probably used a bunch, we should be compatible to them. I guess we should re-introduce last-column-id as optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression) [iceberg-go]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #538:
URL: https://github.com/apache/iceberg-go/issues/538#issuecomment-3213356593
Let me try and reproduce the issue, the current `AddSchema` function does a max between old and newSchema:
```go
func (b *MetadataBuilder) AddSchema(schema *iceberg.Schema) (*MetadataBuilder, error) {
newSchemaID := b.reuseOrCreateNewSchemaID(schema)
if _, err := b.GetSchemaByID(newSchemaID); err == nil {
if b.lastAddedSchemaID == nil || *b.lastAddedSchemaID != newSchemaID {
b.updates = append(b.updates, NewAddSchemaUpdate(schema))
b.lastAddedSchemaID = &newSchemaID
}
return b, nil
}
b.lastColumnId = max(b.lastColumnId, schema.HighestFieldID())
schema.ID = newSchemaID
b.schemaList = append(b.schemaList, schema)
b.updates = append(b.updates, NewAddSchemaUpdate(schema))
b.lastAddedSchemaID = &newSchemaID
return b, nil
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org