You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "alexandre-normand (via GitHub)" <gi...@apache.org> on 2026/08/05 16:49:05 UTC
[PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
alexandre-normand opened a new pull request, #1654:
URL: https://github.com/apache/iceberg-go/pull/1654
We've got a production service that's streaming records into iceberg at _some_ scale and found that one of the lowest hanging fruits in terms of CPU usage (which turned out to be a bottleneck) is the json decoding of iceberg REST catalog responses. Granted, this is exacerbated on our side because we have a long snapshot history and that inflates responses but I thought we might not be the only ones that could benefit from this performance improvement.
The actual change is to swap out `encoding/json` for the drop-in replacement that is [github.com/goccy/go-json](https://github.com/goccy/go-json). We'd probably want to migrate back to `encoding/json` v2 which brings performance improvements but given it's still experimental, `goccy/go-json` is a fair compromise.
You can see the profiling comparison below before/after swapping `encoding/json` for `github.com/goccy/go-json`:
<img width="1715" height="964" alt="performance improvement" src="https://github.com/user-attachments/assets/0724093a-2573-4ed4-848a-2d14eeb8614d" />
With the same resources, we went from 75% to 100% CPU utilization down to 34% with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3760188175
##########
go.mod:
##########
@@ -39,6 +39,7 @@ require (
github.com/docker/docker v28.5.2+incompatible
github.com/geoarrow/geoarrow-go v0.0.0-20260403143023-f54751c3e3a1
github.com/go-sql-driver/mysql v1.10.0
+ github.com/goccy/go-json v0.10.6
Review Comment:
> I'd wait for a release that carries both fixes before pinning, or narrow the swap so we aren't exposed. wdyt?
I think narrowing the scope is actually the path forward here and I'm updating the PR to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#issuecomment-5270776345
> A suggestion that may get you the win sooner: this module already builds on Go 1.26, where stdlib encoding/json/v2 ships with substantial decode-performance improvements and none of the third-party risk. Your benchmark harness is ideally placed to compare it — if it delivers comparable numbers, that's a much easier merge.
A quick note on that until I get to the bigger question (I'll follow-up on the other questions later today): Go 1.26 still has `encoding/json/v2` has an experiment and our org doesn't build production images with experiments in most circumstances. Technically, once 1.27 lands, encoding/json will be using `encoding/json/v2` under the hood so no changes would be needed then to get some decent performance improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#issuecomment-5195381900
I think I prematurely celebrated. It seems like goccy/go-json is triggering a panic in the `TestV3PartitionStatisticsRoundTrip` test.
```
--- FAIL: TestV3PartitionStatisticsRoundTrip (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x1027fddd4]
goroutine 49 [running]:
testing.tRunner.func1.2({0x10555a580, 0x105a88ef0})
/opt/homebrew/opt/go/libexec/src/testing/testing.go:1974 +0x1a0
testing.tRunner.func1()
/opt/homebrew/opt/go/libexec/src/testing/testing.go:1977 +0x318
panic({0x10555a580?, 0x105a88ef0?})
/opt/homebrew/opt/go/libexec/src/runtime/panic.go:860 +0x12c
encoding/json.(*encodeState).marshal.func1()
/opt/homebrew/opt/go/libexec/src/encoding/json/encode.go:339 +0x70
panic({0x10555a580?, 0x105a88ef0?})
/opt/homebrew/opt/go/libexec/src/runtime/panic.go:860 +0x12c
github.com/goccy/go-json/internal/encoder/vm.Run(0x732f6abe4ea0, {0x732f6a9f2400?, 0x0?, 0x400?}, 0x732f6a6e44d0?)
/Users/alex.normand/go/pkg/mod/github.com/goccy/go-json@v0.10.6/internal/encoder/vm/vm.go:26 +0x64
github.com/goccy/go-json.encodeRunCode(0x732f6abe4ea0?, {0x732f6a9f2400?, 0x105b246d0?, 0x732f6a143a50?}, 0x732f6abe4ea0?)
/Users/alex.normand/go/pkg/mod/github.com/goccy/go-json@v0.10.6/encode.go:310 +0x64
github.com/goccy/go-json.encode(0x732f6abe4ea0, {0x105822c80, 0x732f6ad88ad0})
/Users/alex.normand/go/pkg/mod/github.com/goccy/go-json@v0.10.6/encode.go:235 +0x1e0
github.com/goccy/go-json.marshal({0x105822c80, 0x732f6ad88ad0}, {0x0, 0x0, 0x105832a01?})
/Users/alex.normand/go/pkg/mod/github.com/goccy/go-json@v0.10.6/encode.go:150 +0xb0
github.com/goccy/go-json.MarshalWithOption(...)
/Users/alex.normand/go/pkg/mod/github.com/goccy/go-json@v0.10.6/json.go:185
github.com/goccy/go-json.Marshal({0x105822c80?, 0x732f6ad88ad0?})
/Users/alex.normand/go/pkg/mod/github.com/goccy/go-json@v0.10.6/json.go:170 +0x30
github.com/apache/iceberg-go/table.(*metadataV3).MarshalJSON(0x732f6a762900)
/Users/alex.normand/projects/iceberg-go/table/metadata.go:2665 +0x108
encoding/json.marshalerEncoder(0x732f6ab84600, {0x105832ae0?, 0x732f6a762900?, 0x0?}, {0x0?, 0x0?})
/opt/homebrew/opt/go/libexec/src/encoding/json/encode.go:483 +0x9c
encoding/json.(*encodeState).reflectValue(0x732f6ab84600, {0x105832ae0?, 0x732f6a762900?, 0x0?}, {0x0?, 0x0?})
/opt/homebrew/opt/go/libexec/src/encoding/json/encode.go:367 +0x70
encoding/json.(*encodeState).marshal(0x105480480?, {0x105832ae0?, 0x732f6a762900?}, {0xb0?, 0x65?})
/opt/homebrew/opt/go/libexec/src/encoding/json/encode.go:343 +0xac
encoding/json.Marshal({0x105832ae0, 0x732f6a762900})
/opt/homebrew/opt/go/libexec/src/encoding/json/encode.go:209 +0x8c
github.com/apache/iceberg-go/table.TestV3PartitionStatisticsRoundTrip(0x732f6b005b08)
/Users/alex.normand/projects/iceberg-go/table/metadata_internal_test.go:2089 +0x3c0
testing.tRunner(0x732f6b005b08, 0x105874818)
/opt/homebrew/opt/go/libexec/src/testing/testing.go:2036 +0xc4
created by testing.(*T).Run in goroutine 1
/opt/homebrew/opt/go/libexec/src/testing/testing.go:2101 +0x3a8
```
I tried figuring out why and finding a fix for that but no luck so far and I have to switch to some other higher priority work. I'll come back to it at some point but I'll mark this as draft in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3740296235
##########
go.mod:
##########
@@ -39,6 +39,7 @@ require (
github.com/docker/docker v28.5.2+incompatible
github.com/geoarrow/geoarrow-go v0.0.0-20260403143023-f54751c3e3a1
github.com/go-sql-driver/mysql v1.10.0
+ github.com/goccy/go-json v0.10.6
Review Comment:
I'd hold on this pin specifically. goccy v0.10.6 has an open data race in its encode/decode compiler cache (goccy #566) that isn't fixed in any released version — the fix landed on master in July but there's no v0.10.7 to pin. We fan decode out across goroutines (errgroup in `arrow_scanner`, plus the REST catalog from concurrent contexts), so the first-use-of-a-type window is exactly the shape that race hits, and it stays invisible on non-`-race` production builds.
That's on top of the marshal panic I flagged on `metadata.go`, which the PR's own CI already hits (`TestV3PartitionStatisticsRoundTrip` SIGSEGVs inside goccy's encoder). So we've got two independent v0.10.6 defects and nothing stable to upgrade to.
I'd wait for a release that carries both fixes before pinning, or narrow the swap so we aren't exposed. wdyt?
##########
table/metadata.go:
##########
@@ -31,6 +30,8 @@ import (
"strconv"
"time"
+ "github.com/goccy/go-json"
Review Comment:
This is where the CI failure comes from. `metadataV3.MarshalJSON` builds an anonymous struct with an embedded `*Alias` and a sibling `*int64`, hands it to `Marshal`, and goccy v0.10.6 nil-derefs inside `encoder/vm.Run` — that's `TestV3PartitionStatisticsRoundTrip` panicking in this run, and it lines up with the open goccy nested-embedded-omitempty panics (#576/#581/#587, #554). `RESTDataFile`'s six pointer-omitempty fields are the same shape.
The perf goal here is REST decode CPU, and none of the marshal paths benefit from the swap. I'd keep `Marshal`/`MarshalJSON` on `encoding/json` and only route the decode hot paths (`rest.go`, `scan_planning`, `scan_task_decoder`, `catalog/internal/utils`) through goccy. That sidesteps the panic entirely and shrinks the blast radius from ~90 files to the handful that actually get the win.
Separately, goccy normalizes invalid UTF-8 on marshal (replaces with the replacement char) where stdlib passes bytes through — a legacy table with a stray byte in a column name or property would round-trip differently, which is another reason I'd leave marshaling alone. Thoughts?
##########
catalog/internal/utils.go:
##########
@@ -20,9 +20,9 @@ package internal
import (
"compress/gzip"
"context"
- "encoding/json"
"errors"
"fmt"
+ "github.com/goccy/go-json"
Review Comment:
goccy went into the stdlib import group here, so gofumpt fails — this is the red "Audit and Verify" / macOS lint check (it names this file plus `planfake/server.go` and `server_test.go`). Same pattern in a few more (`codec/file_scan_task.go`, the gcs and parquet_files tests).
A blank line before the goccy import to start the third-party group fixes it; running `golangci-lint fmt` at v2.11.4 will do it everywhere at once.
##########
catalog/rest/scan_task_decoder.go:
##########
@@ -20,12 +20,13 @@ package rest
import (
"bytes"
"encoding/hex"
- "encoding/json"
"errors"
"fmt"
"math"
"slices"
+ "github.com/goccy/go-json"
Review Comment:
While we're in the decode paths — `decodeJSONInteger` below calls `UseNumber()`, and goccy v0.10.6 runs `ParseFloat` on every number literal even under `UseNumber` (goccy #555/#599). Within int64 range `Int64()` still parses exact, so I don't think it's a data-loss path, but it's untested. Could we add a case at `MaxInt64` and one past 2^53 to lock the behavior down?
##########
udf/metadata_test.go:
##########
@@ -549,12 +550,12 @@ func TestMetadataValidation(t *testing.T) {
{
"malformed parameter",
func(t *testing.T, m map[string]any) { definition(t, m, 0)["parameters"] = []any{42} },
- nil, "cannot unmarshal number",
+ nil, "invalid character",
Review Comment:
I'd not couple the assertion to goccy's wording here. "invalid character" is goccy's internal syntax-layer phrasing (stdlib classified this as a type error, "cannot unmarshal number"), so it's both weaker — any syntax error now matches — and liable to break on a goccy bump.
I'd assert on the error type instead, `errors.As` into the JSON error or `errors.Is` on `ErrInvalidUDFMetadata`, so we're testing behavior rather than a third-party string. Same applies to the other three rows changed in this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] improve(rest): add load table response json decoding benchmark [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3759908294
##########
table/metadata.go:
##########
@@ -31,6 +30,8 @@ import (
"strconv"
"time"
+ "github.com/goccy/go-json"
Review Comment:
Yes, this is fine. The main reason I did a wholesale swap was to keep things simple and have a cleaner mental model (it's easier to think about iceberg-go using a single library for json decoding). But I'm totally okay with scoping down the goccy/go-json usages where it actually matters most.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#issuecomment-5256714421
> Thanks for this — the goal is a fair one and the swap is mechanically consistent across the tree, which I appreciate.
>
> I'd hold this before merging though, and the reason is the same thing three of us landed on independently: goccy v0.10.6 specifically isn't a safe pin, and the PR's own CI is red on it right now.
>
> The blocking one is the encoder. `metadataV3.MarshalJSON` hands an anonymous struct with an embedded `*Alias` and a sibling `*int64` to goccy, and v0.10.6 nil-derefs inside its encoder — that's `TestV3PartitionStatisticsRoundTrip` panicking with a SIGSEGV in this run, and it matches a cluster of open goccy panics on nested embedded omitempty structs. `RESTDataFile` has the same shape. On top of that goccy has an open data race in its compiler cache (their #566) whose fix isn't in any released version yet, and we decode from goroutines (errgroup fan-out, REST catalog under concurrency), so that race window is one we'd actually hit — silently, on non-`-race` builds.
>
> The thing that makes me most comfortable holding is that there's a clean, narrower version of this change. The perf target is REST decode CPU, and none of the marshal paths benefit — so keeping `Marshal`/`MarshalJSON` on `encoding/json` and routing only the decode hot paths through goccy sidesteps the panic entirely, avoids the UTF-8 normalization divergence on write, and drops the blast radius from ~90 files (CLI, SQL catalog, Puffin, every test) down to the few that get the win.
>
> A few things I'd want settled before merge:
>
> * pin a goccy release that carries both the encoder-panic and the #566 race fixes (or hold until one exists)
> * scope the swap to the REST decode paths; leave marshaling and the non-hot-path packages on stdlib
> * green up the gofumpt lint check (blank line before the goccy import in the six affected files)
> * decouple the `udf/metadata_test.go` assertions from goccy's error strings
> * add a benchmark in-repo (there's precedent with the `*_bench_test.go` files) so the perf win is reproducible, plus a large-int64 decode case
>
> Once those are addressed I'm happy to take another pass and approve.
@laskoviymishka I think I addressed all the things you had on your list except this one:
> * pin a goccy release that carries both the encoder-panic and the #566 race fixes (or hold until one exists)
I could update to use the latest commit and not block on a new tagged version.
I think it's likely that we might just want to go with `encoding/json/v2` instead of waiting for a tagged release but I would like to get better performance without having to wait for go 1.27. I wouldn't be offended if you just declined the PR and I could just use this change on our fork until go 1.27 lands and, hopefully, leads to better performance equal or better than goccy/go-json.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3760275980
##########
catalog/rest/scan_task_decoder.go:
##########
@@ -20,12 +20,13 @@ package rest
import (
"bytes"
"encoding/hex"
- "encoding/json"
"errors"
"fmt"
"math"
"slices"
+ "github.com/goccy/go-json"
Review Comment:
That makes sense. I added two test cases to `TestDecodePartitionLiteralCoversPrimitiveWireTypes` that should cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] improve(rest): add load table response json decoding benchmark [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#issuecomment-5295237570
>This is a much better scope — thanks for pivoting. The benchmark harness and the int64 boundary coverage for look good.\n\nOne leftover nit: the last benchmark case is named but its func now uses , so it'''s a duplicate of the previous case with a misleading label. Please either drop it or rename it — fine to fix before merge.
oops, sorry about that. I just pushed a commit to delete the duplicate/wrongly named test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] improve(rest): add load table response json decoding benchmark [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#issuecomment-5273491421
I decided to scrap the plans for goccy/go-json given that we might be able to negate the poor performance impact of encoding/json from the [recently merged PR](https://github.com/apache/iceberg-go/pull/1749) from @technicolorbeat (we don't need to get the full snapshot list when loading tables and this is the bulk of the json that we have to decode).
I could have either completely closed this PR _or_ trimmed it down to only the benchmark that would be scoped _only_ to encoding/json benchmarking. I decided to do the latter since it was easy enough to do and you can make the call whether this can/should be merged or not worth it.
I could see value in keeping the benchmark around so that when 1.27 lands with encoding/json, we can review benchmark numbers before/after the golang upgrade and actually see how performance does improve.
Thinking ahead, I don't know if moving to using `encoding/json/v2` would be beneficial then vs just using `encoding/json` which will be backed with `encoding/json/v2` using v1 compatibility options. I haven't reviewed the details but it should be that just upgrading to 1.27 will be enough to bring performance benefits without requiring code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] improve(rest): add load table response json decoding benchmark [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3779436072
##########
catalog/rest/load_table_bench_test.go:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package rest
+
+import (
+ "encoding/json"
+ "fmt"
+ "testing"
+
+ "github.com/apache/iceberg-go/table"
+ "github.com/google/uuid"
+ "github.com/stretchr/testify/require"
+)
+
+// BenchmarkDecodeTableMetadata benchmarks the json decoding of LoadTable responses. Because responses can get big
+// with tables with a long snapshot history, the impact of the json decoding performance can be significant in
+// higher-throughput workloads. This benchmark can be extended to experiment with other json decoder for
+// performance comparisons.
+func BenchmarkDecodeTableMetadata(b *testing.B) {
+ snapshotCounts := []struct {
+ name string
+ snapshotCount int64
+ decode func(body []byte, v any) error
+ }{
+ {
+ name: "1 snapshot, encoding/json",
+ snapshotCount: 1,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10 snapshots, encoding/json",
+ snapshotCount: 10,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "100 snapshots, encoding/json",
+ snapshotCount: 100,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "1000 snapshots, encoding/json",
+ snapshotCount: 1000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, encoding/json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, goccy/go-json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ }
+
+ for _, tc := range snapshotCounts {
+ b.Run(tc.name, func(b *testing.B) {
+ body := makeTableResponseWithSnapshots(tc.snapshotCount)
+
+ b.ResetTimer()
+ b.ReportAllocs()
+
+ for i := 0; i < b.N; i++ {
+ var resp loadTableResponse
+ err := tc.decode(body, &resp)
+ require.NoError(b, err)
+ }
+ })
+ }
+}
+
+// makeTableResponseWithSnapshots formats a LoadTable json body with a specified number of snapshots. Snapshots are used
+// here to exercise different response size profiles as it's one of the common sources for large responses.
+func makeTableResponseWithSnapshots(snapshotCount int64) []byte {
+ snapshots := make([]table.Snapshot, 0, snapshotCount)
+ snapshotLogEntries := make([]table.SnapshotLogEntry, 0, snapshotCount)
+ schemaID := 0
+ var snapshotTimestamp int64
+ var snapshotID int64
+ for i := int64(0); i < snapshotCount; i++ {
+ var parentID *int64
+ if i > 0 {
+ parentID = &i
Review Comment:
`parentID = &i` takes the address of the loop counter, not a copy. This is a C-style for loop, so Go 1.22's per-iteration capture doesn't apply and every snapshot shares the one `i`. After the loop `i == snapshotCount`, so every non-first snapshot marshals with `parent-snapshot-id: <snapshotCount>`, an ID that isn't in the list.
Doesn't move the decode timing, but the fixture stops being a valid snapshot chain, which bites anyone who copies this helper for a correctness test. `prev := i - 1; parentID = &prev` fixes it.
##########
catalog/rest/load_table_bench_test.go:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package rest
+
+import (
+ "encoding/json"
+ "fmt"
+ "testing"
+
+ "github.com/apache/iceberg-go/table"
+ "github.com/google/uuid"
+ "github.com/stretchr/testify/require"
+)
+
+// BenchmarkDecodeTableMetadata benchmarks the json decoding of LoadTable responses. Because responses can get big
+// with tables with a long snapshot history, the impact of the json decoding performance can be significant in
+// higher-throughput workloads. This benchmark can be extended to experiment with other json decoder for
+// performance comparisons.
+func BenchmarkDecodeTableMetadata(b *testing.B) {
+ snapshotCounts := []struct {
+ name string
+ snapshotCount int64
+ decode func(body []byte, v any) error
+ }{
+ {
+ name: "1 snapshot, encoding/json",
+ snapshotCount: 1,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10 snapshots, encoding/json",
+ snapshotCount: 10,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "100 snapshots, encoding/json",
+ snapshotCount: 100,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "1000 snapshots, encoding/json",
+ snapshotCount: 1000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, encoding/json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, goccy/go-json",
Review Comment:
This row is labeled `goccy/go-json` but its decode closure calls `encoding/json`'s Unmarshal, identical to the `encoding/json` row above it, so the benchmark reports two 10000-snapshot numbers that are the same decoder under two names. The whole point of the harness is comparing decoders, so a mislabeled slot is worse than no slot: anyone reading the output concludes goccy is a wash, on fake data.
I'd drop this entry until there's a real second decoder to put here. Matt's suggestion of `encoding/json/v2` on 1.26 is the natural thing to slot in: a real comparison and no third-party risk. wdyt?
##########
catalog/rest/load_table_bench_test.go:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package rest
+
+import (
+ "encoding/json"
+ "fmt"
+ "testing"
+
+ "github.com/apache/iceberg-go/table"
+ "github.com/google/uuid"
+ "github.com/stretchr/testify/require"
+)
+
+// BenchmarkDecodeTableMetadata benchmarks the json decoding of LoadTable responses. Because responses can get big
+// with tables with a long snapshot history, the impact of the json decoding performance can be significant in
+// higher-throughput workloads. This benchmark can be extended to experiment with other json decoder for
+// performance comparisons.
+func BenchmarkDecodeTableMetadata(b *testing.B) {
+ snapshotCounts := []struct {
+ name string
+ snapshotCount int64
+ decode func(body []byte, v any) error
+ }{
+ {
+ name: "1 snapshot, encoding/json",
+ snapshotCount: 1,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10 snapshots, encoding/json",
+ snapshotCount: 10,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "100 snapshots, encoding/json",
+ snapshotCount: 100,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "1000 snapshots, encoding/json",
+ snapshotCount: 1000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, encoding/json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, goccy/go-json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ }
+
+ for _, tc := range snapshotCounts {
+ b.Run(tc.name, func(b *testing.B) {
+ body := makeTableResponseWithSnapshots(tc.snapshotCount)
+
+ b.ResetTimer()
+ b.ReportAllocs()
+
+ for i := 0; i < b.N; i++ {
+ var resp loadTableResponse
+ err := tc.decode(body, &resp)
+ require.NoError(b, err)
Review Comment:
`require.NoError` runs testify's reflection/formatting machinery every iteration, and for the 1- and 10-snapshot cases that's a real fraction of the measured time. `if err != nil { b.Fatal(err) }` keeps the timed loop clean.
While we're in here: `ReportAllocs` conventionally goes before `ResetTimer`, and since the module's on 1.25 you could drop both the `b.N` loop and `ResetTimer` for `for b.Loop() { ... }`, which handles the reset itself.
##########
catalog/rest/scan_task_decoder_test.go:
##########
@@ -482,34 +484,42 @@ func TestDecodePartitionLiteralCoversPrimitiveWireTypes(t *testing.T) {
decimalType := iceberg.DecimalTypeOf(9, 2)
fixedType := iceberg.FixedTypeOf(4)
tests := []struct {
- name string
- raw string
- typ iceberg.Type
- want iceberg.Literal
+ name string
+ raw string
+ typ iceberg.Type
+ want iceberg.Literal
+ expectedErr error
}{
- {"boolean", `true`, iceberg.PrimitiveTypes.Bool, iceberg.BoolLiteral(true)},
- {"int", `2147483647`, iceberg.PrimitiveTypes.Int32, iceberg.Int32Literal(2147483647)},
- {"long above JSON exact float range", `9007199254740993`, iceberg.PrimitiveTypes.Int64, iceberg.Int64Literal(9007199254740993)},
- {"float", `1.25`, iceberg.PrimitiveTypes.Float32, iceberg.Float32Literal(1.25)},
- {"double", `1.25`, iceberg.PrimitiveTypes.Float64, iceberg.Float64Literal(1.25)},
- {"string", `"hello"`, iceberg.PrimitiveTypes.String, iceberg.StringLiteral("hello")},
- {"date", `"2026-07-17"`, iceberg.PrimitiveTypes.Date, mustLiteral(t, "2026-07-17", iceberg.PrimitiveTypes.Date)},
- {"time", `"10:15:30.123456"`, iceberg.PrimitiveTypes.Time, mustLiteral(t, "10:15:30.123456", iceberg.PrimitiveTypes.Time)},
- {"timestamp", `"2026-07-17T10:15:30.123456"`, iceberg.PrimitiveTypes.Timestamp, mustLiteral(t, "2026-07-17T10:15:30.123456", iceberg.PrimitiveTypes.Timestamp)},
- {"timestamptz", `"2026-07-17T10:15:30.123456+00:00"`, iceberg.PrimitiveTypes.TimestampTz, mustLiteral(t, "2026-07-17T10:15:30.123456+00:00", iceberg.PrimitiveTypes.TimestampTz)},
- {"timestamp nanos", `"2026-07-17T10:15:30.123456789"`, iceberg.PrimitiveTypes.TimestampNs, mustLiteral(t, "2026-07-17T10:15:30.123456789", iceberg.PrimitiveTypes.TimestampNs)},
- {"timestamptz nanos", `"2026-07-17T10:15:30.123456789+00:00"`, iceberg.PrimitiveTypes.TimestampTzNs, mustLiteral(t, "2026-07-17T10:15:30.123456789+00:00", iceberg.PrimitiveTypes.TimestampTzNs)},
- {"decimal", `"12.34"`, decimalType, mustLiteral(t, "12.34", decimalType)},
- {"uuid", `"f79c3e09-677c-4bbd-a479-3f349cb785e7"`, iceberg.PrimitiveTypes.UUID, mustLiteral(t, "f79c3e09-677c-4bbd-a479-3f349cb785e7", iceberg.PrimitiveTypes.UUID)},
- {"fixed", `"78797A21"`, fixedType, iceberg.FixedLiteral([]byte("xyz!"))},
- {"binary", `"00FF10"`, iceberg.PrimitiveTypes.Binary, iceberg.BinaryLiteral([]byte{0, 255, 16})},
+ {"boolean", `true`, iceberg.PrimitiveTypes.Bool, iceberg.BoolLiteral(true), nil},
+ {"int", `2147483647`, iceberg.PrimitiveTypes.Int32, iceberg.Int32Literal(2147483647), nil},
+ {"long above JSON exact float range", `9007199254740993`, iceberg.PrimitiveTypes.Int64, iceberg.Int64Literal(9007199254740993), nil},
+ {"max int64", `9223372036854775807`, iceberg.PrimitiveTypes.Int64, iceberg.Int64Literal(math.MaxInt64), nil},
+ {"above max int64", `9223372036854775808`, iceberg.PrimitiveTypes.Int64, iceberg.Int64Literal(0), strconv.ErrRange},
Review Comment:
The `want` on this row is dead: the error branch returns before `got.Equals(tt.want)` runs, so `iceberg.Int64Literal(0)` is never checked and reads as if we assert a zero literal on overflow when we don't. `nil` says "unused" unambiguously since `want` is an interface. Or, if you do want to pin the return value on the error path, assert `got` inside the `expectedErr` branch, either's fine, just not a placeholder that looks load-bearing.
##########
catalog/rest/load_table_bench_test.go:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package rest
+
+import (
+ "encoding/json"
+ "fmt"
+ "testing"
+
+ "github.com/apache/iceberg-go/table"
+ "github.com/google/uuid"
+ "github.com/stretchr/testify/require"
+)
+
+// BenchmarkDecodeTableMetadata benchmarks the json decoding of LoadTable responses. Because responses can get big
+// with tables with a long snapshot history, the impact of the json decoding performance can be significant in
+// higher-throughput workloads. This benchmark can be extended to experiment with other json decoder for
+// performance comparisons.
+func BenchmarkDecodeTableMetadata(b *testing.B) {
+ snapshotCounts := []struct {
+ name string
+ snapshotCount int64
+ decode func(body []byte, v any) error
+ }{
+ {
+ name: "1 snapshot, encoding/json",
+ snapshotCount: 1,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10 snapshots, encoding/json",
+ snapshotCount: 10,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "100 snapshots, encoding/json",
+ snapshotCount: 100,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "1000 snapshots, encoding/json",
+ snapshotCount: 1000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, encoding/json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, goccy/go-json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ }
+
+ for _, tc := range snapshotCounts {
+ b.Run(tc.name, func(b *testing.B) {
+ body := makeTableResponseWithSnapshots(tc.snapshotCount)
+
+ b.ResetTimer()
+ b.ReportAllocs()
+
+ for i := 0; i < b.N; i++ {
+ var resp loadTableResponse
+ err := tc.decode(body, &resp)
+ require.NoError(b, err)
+ }
+ })
+ }
+}
+
+// makeTableResponseWithSnapshots formats a LoadTable json body with a specified number of snapshots. Snapshots are used
+// here to exercise different response size profiles as it's one of the common sources for large responses.
+func makeTableResponseWithSnapshots(snapshotCount int64) []byte {
+ snapshots := make([]table.Snapshot, 0, snapshotCount)
+ snapshotLogEntries := make([]table.SnapshotLogEntry, 0, snapshotCount)
+ schemaID := 0
+ var snapshotTimestamp int64
+ var snapshotID int64
+ for i := int64(0); i < snapshotCount; i++ {
+ var parentID *int64
+ if i > 0 {
+ parentID = &i
+ }
+ snapshotTimestamp = 1785448901408 + i
+ snapshotID = i
+
+ snapshots = append(snapshots, table.Snapshot{
+ SnapshotID: snapshotID,
+ ParentSnapshotID: parentID,
+ SequenceNumber: i,
+ TimestampMs: snapshotTimestamp,
+ ManifestList: fmt.Sprintf("s3://warehouse/database/table/metadata/snap-%s.avro", uuid.NewString()),
+ Summary: &table.Summary{
+ Operation: "append",
+ Properties: map[string]string{
+ "spark.app.id": "local-1646787004168",
+ "added-data-files": "1",
+ "added-records": "1",
+ "added-files-size": "697",
+ "changed-partition-count": "1",
+ "total-records": "1",
+ "total-files-size": "697",
+ "total-data-files": "1",
+ "total-delete-files": "0",
+ "total-position-deletes": "0",
+ "total-equality-deletes": "0",
+ },
+ },
+ SchemaID: &schemaID,
+ })
+ snapshotLogEntries = append(snapshotLogEntries, table.SnapshotLogEntry{
+ SnapshotID: i,
+ TimestampMs: snapshotTimestamp,
+ })
+ }
+ snapshotsJson, err := json.Marshal(snapshots)
+ if err != nil {
+ panic(fmt.Errorf("failed to generate load table response: %w", err))
+ }
+
+ snapshotsLogEntriesJson, err := json.Marshal(snapshotLogEntries)
+ if err != nil {
+ panic(fmt.Errorf("failed to generate load table response: %w", err))
+ }
+
+ return []byte(fmt.Sprintf(`{
+ "metadata-location": "s3://warehouse/database/table/metadata/00001-5f2f8166-244c-4eae-ac36-384ecdec81fc.gz.metadata.json",
+ "metadata": {
+ "format-version": 1,
Review Comment:
Small realism thing: the fixture says format-version 1 but each snapshot carries a `sequence-number` (`Snapshot.SequenceNumber` has no `omitempty` and you set it to `i`), plus it uses `schemas`/`current-schema-id`, which are V2 shapes. A real V1 LoadTable response wouldn't have per-snapshot sequence numbers, so the decode is chewing tokens it'd never see in the wild. Since the body's already V2-shaped, I'd just bump this to `"format-version": 2`. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] improve(rest): add load table response json decoding benchmark [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3779436063
##########
catalog/rest/load_table_bench_test.go:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package rest
+
+import (
+ "encoding/json"
+ "fmt"
+ "testing"
+
+ "github.com/apache/iceberg-go/table"
+ "github.com/google/uuid"
+ "github.com/stretchr/testify/require"
+)
+
+// BenchmarkDecodeTableMetadata benchmarks the json decoding of LoadTable responses. Because responses can get big
+// with tables with a long snapshot history, the impact of the json decoding performance can be significant in
+// higher-throughput workloads. This benchmark can be extended to experiment with other json decoder for
+// performance comparisons.
+func BenchmarkDecodeTableMetadata(b *testing.B) {
+ snapshotCounts := []struct {
+ name string
+ snapshotCount int64
+ decode func(body []byte, v any) error
+ }{
+ {
+ name: "1 snapshot, encoding/json",
+ snapshotCount: 1,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10 snapshots, encoding/json",
+ snapshotCount: 10,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "100 snapshots, encoding/json",
+ snapshotCount: 100,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "1000 snapshots, encoding/json",
+ snapshotCount: 1000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, encoding/json",
+ snapshotCount: 10000,
+ decode: func(body []byte, v any) error {
+ return json.Unmarshal(body, v)
+ },
+ },
+ {
+ name: "10000 snapshots, goccy/go-json",
Review Comment:
This row is labeled `goccy/go-json` but its decode closure calls `encoding/json`'s Unmarshal, identical to the `encoding/json` row above it, so the benchmark reports two 10000-snapshot numbers that are the same decoder under two names. The whole point of the harness is comparing decoders, so a mislabeled slot is worse than no slot: anyone reading the output concludes goccy is a wash, on fake data.
I'd drop this entry until there's a real second decoder to put here. Matt's suggestion of `encoding/json/v2` on 1.26 is the natural thing to slot in: a real comparison and no third-party risk. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#issuecomment-5271582243
Fair point on `GOEXPERIMENT=jsonv2` — agreed that's not production-viable until it graduates (1.27 hopefully). To be clear, json/v2 was only an optional suggestion; the actual blocker from my review is solely the unreleased fix for goccy/go-json#566, since the decode-compiler race sits exactly on the REST catalog's concurrent path. Happy to re-review as soon as either a goccy release carries that fix or jsonv2 graduates — whichever comes first. Looking forward to your follow-up on the rest.
<sub>_Posted by an AI-assisted tool on behalf of maintainer `zeroshade`._</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [improve] Use goccy/go-json instead of encoding/json for faster catalog response decoding [iceberg-go]
Posted by "alexandre-normand (via GitHub)" <gi...@apache.org>.
alexandre-normand commented on code in PR #1654:
URL: https://github.com/apache/iceberg-go/pull/1654#discussion_r3759918890
##########
udf/metadata_test.go:
##########
@@ -549,12 +550,12 @@ func TestMetadataValidation(t *testing.T) {
{
"malformed parameter",
func(t *testing.T, m map[string]any) { definition(t, m, 0)["parameters"] = []any{42} },
- nil, "cannot unmarshal number",
+ nil, "invalid character",
Review Comment:
I've reverted all of those changes. I agree about the assertion behavior but I wanted to keep things focused on the improvement so I decided to leave out revisiting this as part of the `catalog/rest` scoped down version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org