You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ze...@apache.org on 2026/05/19 03:10:51 UTC
(arrow-go) branch main updated: fix(arrow/array): preserve large integer precision in JSON decoding (#816)
This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-go.git
The following commit(s) were added to refs/heads/main by this push:
new fbc39237 fix(arrow/array): preserve large integer precision in JSON decoding (#816)
fbc39237 is described below
commit fbc3923778d2f20dcb59b9e4a3f69841dbfdc5a2
Author: Matt Topol <zo...@gmail.com>
AuthorDate: Mon May 18 23:10:45 2026 -0400
fix(arrow/array): preserve large integer precision in JSON decoding (#816)
## Summary
Fixes #804.
JSON decoding in `arrow/array` silently corrupted integer values larger
than 2^53 because Go's `encoding/json` decodes numeric tokens as
`float64` by default, losing precision for very large integers. This PR
makes `UseNumber` the unconditional default across `FromJSON`,
`RecordFromJSON`, and `NewJSONReader` so full `int64` precision is
preserved without requiring callers to opt in.
## Changes
- **`RecordBuilder` gains `UnmarshalOne` and `Unmarshal`** so
caller-configured `json.Decoder` options propagate through nested field
decoding. `RecordFromJSON` now calls `bldr.UnmarshalOne(dec)` instead of
`dec.Decode(bldr)`, avoiding the bytes-roundtrip through
`json.Unmarshaler` that previously dropped the outer decoder
configuration (root cause noted in [this
comment](https://github.com/apache/arrow-go/issues/804#issuecomment-4454244067)).
- **All builder `UnmarshalJSON([]byte)` methods now enable `UseNumber`**
on their internal decoder so large integers also survive the standard
`json.Unmarshal(data, &builder)` path.
- **String-encoded integers now work for time-based types.** `Duration`,
`Timestamp`, `Time32`, `Time64`, `Date32`, and `Date64` builders accept
stringified integers (e.g. `"9223372036854775807"`) as a fallback before
their format-specific parsers — addressing the second snippet from the
issue. This matches how `Int64Builder` and the decimal builders already
behaved.
- **`WithUseNumber` and `WithUseNumberJSONReader` are deprecated**
(marked with `Deprecated:` per pkg.go.dev convention) since `UseNumber`
is now always enabled. Both options remain functional no-ops for
backward compatibility.
## Test Plan
15 new tests added in `arrow/array/util_test.go` and
`arrow/array/json_reader_test.go` covering:
- `RecordFromJSON` and `JSONReader` round-tripping `MaxInt64`/`MinInt64`
with and without the deprecated options.
- `RecordBuilder.UnmarshalJSON` and `UnmarshalOne` directly preserving
large integers.
- Stringified integer round-trip for `Duration`, `Timestamp`, `Time32`,
`Time64`, `Date32`, `Date64`.
- Regression: invalid duration strings (`"abc"`) still error;
format-style duration strings (`"3h2m0.5s"`) still parse.
Verified passing locally:
- `./arrow/array/...` (full suite)
- `./arrow/compute/...` (largest blast-radius consumer of
`WithUseNumber` — 49 test sites)
- `./arrow/ipc/...`, `./arrow/scalar/...`, `./arrow/extensions/...`,
`./arrow/encoded/...`, `./arrow/util/...`
- `./parquet/file/...` (the `./parquet/pqarrow/...` package fails
identically on `main` due to a missing `PARQUET_TEST_DATA` env var —
pre-existing, unrelated)
All `lsp_diagnostics` clean across the 23 changed files.
## Behavior Change Notes
The only observable behavior change for existing callers is: large
integer values that previously decoded to a wrong `int64` (via float64
round-trip) now decode to the correct value. Existing tests in
`arrow/compute` continue to pass, including those that explicitly opt in
via `WithUseNumber` and the ones that don't. No production callers in
this repo break (verified across `flight`, `compute`, `parquet`, `ipc`).
---
arrow/array/binarybuilder.go | 2 +
arrow/array/decimal.go | 1 +
arrow/array/dictionary.go | 1 +
arrow/array/encoded.go | 1 +
arrow/array/fixed_size_list.go | 1 +
arrow/array/fixedsize_binarybuilder.go | 1 +
arrow/array/float16_builder.go | 1 +
arrow/array/interval.go | 3 +
arrow/array/json_reader.go | 4 +-
arrow/array/json_reader_test.go | 25 ++
arrow/array/list.go | 2 +
arrow/array/map.go | 1 +
arrow/array/null.go | 1 +
arrow/array/numericbuilder.gen.go | 469 ++++++++++++++++++++++------
arrow/array/numericbuilder.gen.go.tmpl | 199 ++++++++++--
arrow/array/numericbuilder.gen_test.go | 45 ++-
arrow/array/numericbuilder.gen_test.go.tmpl | 6 +-
arrow/array/record.go | 47 ++-
arrow/array/string.go | 3 +
arrow/array/struct.go | 1 +
arrow/array/timestamp.go | 11 +
arrow/array/union.go | 2 +
arrow/array/util.go | 25 +-
arrow/array/util_test.go | 261 ++++++++++++++++
arrow/compute/arithmetic_test.go | 30 +-
arrow/compute/cast_test.go | 34 +-
arrow/compute/scalar_compare_test.go | 20 +-
arrow/compute/scalar_set_lookup_test.go | 2 +-
arrow/compute/vector_selection_test.go | 20 +-
arrow/doc.go | 2 +-
arrow/flight/flightsql/example/type_info.go | 2 +-
31 files changed, 1007 insertions(+), 216 deletions(-)
diff --git a/arrow/array/binarybuilder.go b/arrow/array/binarybuilder.go
index 537ec933..5b881a84 100644
--- a/arrow/array/binarybuilder.go
+++ b/arrow/array/binarybuilder.go
@@ -375,6 +375,7 @@ func (b *BinaryBuilder) Unmarshal(dec *json.Decoder) error {
func (b *BinaryBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -669,6 +670,7 @@ func (b *BinaryViewBuilder) Unmarshal(dec *json.Decoder) error {
func (b *BinaryViewBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/decimal.go b/arrow/array/decimal.go
index 993242b9..84689c95 100644
--- a/arrow/array/decimal.go
+++ b/arrow/array/decimal.go
@@ -387,6 +387,7 @@ func (b *baseDecimalBuilder[T]) Unmarshal(dec *json.Decoder) error {
func (b *baseDecimalBuilder[T]) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/dictionary.go b/arrow/array/dictionary.go
index 8f9fe245..38a43f77 100644
--- a/arrow/array/dictionary.go
+++ b/arrow/array/dictionary.go
@@ -700,6 +700,7 @@ func (b *dictionaryBuilder) IsNull(i int) bool { return b.idxBuilder.IsNull(i) }
func (b *dictionaryBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/encoded.go b/arrow/array/encoded.go
index 85432a13..8d628ffc 100644
--- a/arrow/array/encoded.go
+++ b/arrow/array/encoded.go
@@ -520,6 +520,7 @@ func (b *RunEndEncodedBuilder) Unmarshal(dec *json.Decoder) error {
// UnmarshalJSON can't be used in conjunction with AppendValueFromString (as it calls UnmarshalOne)
func (b *RunEndEncodedBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/fixed_size_list.go b/arrow/array/fixed_size_list.go
index 69cb67b2..d382ebe9 100644
--- a/arrow/array/fixed_size_list.go
+++ b/arrow/array/fixed_size_list.go
@@ -372,6 +372,7 @@ func (b *FixedSizeListBuilder) Unmarshal(dec *json.Decoder) error {
func (b *FixedSizeListBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/fixedsize_binarybuilder.go b/arrow/array/fixedsize_binarybuilder.go
index ca5a10c3..b6745be6 100644
--- a/arrow/array/fixedsize_binarybuilder.go
+++ b/arrow/array/fixedsize_binarybuilder.go
@@ -244,6 +244,7 @@ func (b *FixedSizeBinaryBuilder) Unmarshal(dec *json.Decoder) error {
func (b *FixedSizeBinaryBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/float16_builder.go b/arrow/array/float16_builder.go
index 60cc91b7..25e8f4a9 100644
--- a/arrow/array/float16_builder.go
+++ b/arrow/array/float16_builder.go
@@ -251,6 +251,7 @@ func (b *Float16Builder) Unmarshal(dec *json.Decoder) error {
// be silently truncated.
func (b *Float16Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/interval.go b/arrow/array/interval.go
index 2900a592..2c029c25 100644
--- a/arrow/array/interval.go
+++ b/arrow/array/interval.go
@@ -330,6 +330,7 @@ func (b *MonthIntervalBuilder) Unmarshal(dec *json.Decoder) error {
// value that will be added to the builder.
func (b *MonthIntervalBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -631,6 +632,7 @@ func (b *DayTimeIntervalBuilder) Unmarshal(dec *json.Decoder) error {
// with the values expected to be objects of the form {"days": #, "milliseconds": #}
func (b *DayTimeIntervalBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -936,6 +938,7 @@ func (b *MonthDayNanoIntervalBuilder) Unmarshal(dec *json.Decoder) error {
// {"months": #, "days": #, "nanoseconds": #}
func (b *MonthDayNanoIntervalBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/json_reader.go b/arrow/array/json_reader.go
index d9ef5ed7..6aea688d 100644
--- a/arrow/array/json_reader.go
+++ b/arrow/array/json_reader.go
@@ -103,6 +103,8 @@ func NewJSONReader(r io.Reader, schema *arrow.Schema, opts ...Option) *JSONReade
o(rr)
}
+ rr.r.UseNumber()
+
if rr.mem == nil {
rr.mem = memory.DefaultAllocator
}
@@ -166,7 +168,7 @@ func (r *JSONReader) Next() bool {
}
func (r *JSONReader) readNext() bool {
- r.err = r.r.Decode(r.bldr)
+ r.err = r.bldr.UnmarshalOne(r.r)
if r.err != nil {
r.done = true
if errors.Is(r.err, io.EOF) {
diff --git a/arrow/array/json_reader_test.go b/arrow/array/json_reader_test.go
index d309b0c3..fbf83428 100644
--- a/arrow/array/json_reader_test.go
+++ b/arrow/array/json_reader_test.go
@@ -27,6 +27,7 @@ import (
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/internal/json"
"github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
)
const jsondata = `
@@ -246,6 +247,30 @@ func jsonArrayToNDJSON(data []byte) ([]byte, error) {
return ndjson.Bytes(), nil
}
+func TestJSONReaderLargeInt64(t *testing.T) {
+ schema := arrow.NewSchema([]arrow.Field{
+ {Name: "a", Type: arrow.PrimitiveTypes.Int64},
+ }, nil)
+
+ mem := memory.NewCheckedAllocator(memory.NewGoAllocator())
+ defer mem.AssertSize(t, 0)
+
+ const ndjson = "{\"a\": 9223372036854775807}\n{\"a\": -9223372036854775808}\n"
+ rdr := array.NewJSONReader(strings.NewReader(ndjson), schema,
+ array.WithAllocator(mem), array.WithChunk(-1))
+ defer rdr.Release()
+
+ assert.True(t, rdr.Next())
+ rec := rdr.RecordBatch()
+ require.NotNil(t, rec)
+ assert.NoError(t, rdr.Err())
+ assert.EqualValues(t, 2, rec.NumRows())
+
+ col := rec.Column(0).(*array.Int64)
+ assert.EqualValues(t, int64(9223372036854775807), col.Value(0))
+ assert.EqualValues(t, int64(-9223372036854775808), col.Value(1))
+}
+
func BenchmarkRecordFromJSON(b *testing.B) {
schema := arrow.NewSchema([]arrow.Field{
{Name: "id", Type: arrow.PrimitiveTypes.Int64},
diff --git a/arrow/array/list.go b/arrow/array/list.go
index d87cc266..d72887bb 100644
--- a/arrow/array/list.go
+++ b/arrow/array/list.go
@@ -627,6 +627,7 @@ func (b *baseListBuilder) Unmarshal(dec *json.Decoder) error {
func (b *baseListBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -1415,6 +1416,7 @@ func (b *baseListViewBuilder) Unmarshal(dec *json.Decoder) error {
func (b *baseListViewBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/map.go b/arrow/array/map.go
index da9a150b..71d4d138 100644
--- a/arrow/array/map.go
+++ b/arrow/array/map.go
@@ -342,6 +342,7 @@ func (b *MapBuilder) Unmarshal(dec *json.Decoder) error {
func (b *MapBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/null.go b/arrow/array/null.go
index 02ea12eb..8f8f5805 100644
--- a/arrow/array/null.go
+++ b/arrow/array/null.go
@@ -201,6 +201,7 @@ func (b *NullBuilder) Unmarshal(dec *json.Decoder) error {
func (b *NullBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/numericbuilder.gen.go b/arrow/array/numericbuilder.gen.go
index aa2f628d..b54f3e9d 100644
--- a/arrow/array/numericbuilder.gen.go
+++ b/arrow/array/numericbuilder.gen.go
@@ -21,6 +21,7 @@ package array
import (
"bytes"
"fmt"
+ "math"
"reflect"
"strconv"
"strings"
@@ -218,10 +219,11 @@ func (b *Int64Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any int
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseInt(v, 10, 8*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -230,16 +232,33 @@ func (b *Int64Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(int64(fval)) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(int64(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(int64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := int64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(int64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -252,10 +271,13 @@ func (b *Int64Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(int64(v))
case json.Number:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON like 1.5 or 128 (for int8) is surfaced as
+ // UnmarshalTypeError rather than silently truncated or wrapped.
f, err := strconv.ParseInt(v.String(), 10, 8*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -264,16 +286,33 @@ func (b *Int64Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(int64(fval)) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(int64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(int64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := int64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(int64(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -306,6 +345,7 @@ func (b *Int64Builder) Unmarshal(dec *json.Decoder) error {
func (b *Int64Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -503,10 +543,11 @@ func (b *Uint64Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any uint
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v, 10, 8*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -515,16 +556,33 @@ func (b *Uint64Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint64(0)) {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint64(0))+1 {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(uint64(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(uint64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(uint64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -537,10 +595,14 @@ func (b *Uint64Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(uint64(v))
case json.Number:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON is surfaced as UnmarshalTypeError rather than silently
+ // coerced. Range-check before any uint conversion to avoid undefined
+ // behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v.String(), 10, 8*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -549,16 +611,36 @@ func (b *Uint64Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint64(0)) {
+ // Boundary is max+1: for uint64, valid range is [0, max]; reject anything >= max+1.
+ // For uint64, float64(^uint64(0)) already rounds up to 2^64, so this naturally
+ // rejects the exact 2^64 boundary that the looser `>` check missed.
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint64(0))+1 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(uint64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(uint64(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(uint64(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -591,6 +673,7 @@ func (b *Uint64Builder) Unmarshal(dec *json.Decoder) error {
func (b *Uint64Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -832,6 +915,7 @@ func (b *Float64Builder) Unmarshal(dec *json.Decoder) error {
func (b *Float64Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -1029,10 +1113,11 @@ func (b *Int32Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any int
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseInt(v, 10, 4*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -1041,18 +1126,25 @@ func (b *Int32Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
minVal := float64(int64(-1) << (4*8 - 1))
maxVal := float64(int64(1) << (4*8 - 1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(int32(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ truncated := int64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(int32(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -1065,10 +1157,13 @@ func (b *Int32Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(int32(v))
case json.Number:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON like 1.5 or 128 (for int8) is surfaced as
+ // UnmarshalTypeError rather than silently truncated or wrapped.
f, err := strconv.ParseInt(v.String(), 10, 4*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -1077,18 +1172,25 @@ func (b *Int32Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
minVal := float64(int64(-1) << (4*8 - 1))
maxVal := float64(int64(1) << (4*8 - 1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(int32(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := int64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(int32(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -1121,6 +1223,7 @@ func (b *Int32Builder) Unmarshal(dec *json.Decoder) error {
func (b *Int32Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -1318,10 +1421,11 @@ func (b *Uint32Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any uint
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v, 10, 4*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -1330,16 +1434,23 @@ func (b *Uint32Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint32(0)) {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint32(0))+1 {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(uint32(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(uint32(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -1352,10 +1463,14 @@ func (b *Uint32Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(uint32(v))
case json.Number:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON is surfaced as UnmarshalTypeError rather than silently
+ // coerced. Range-check before any uint conversion to avoid undefined
+ // behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v.String(), 10, 4*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -1364,16 +1479,26 @@ func (b *Uint32Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint32(0)) {
+ // Boundary is max+1: for uint32, valid range is [0, max]; reject anything >= max+1.
+ // For uint64, float64(^uint64(0)) already rounds up to 2^64, so this naturally
+ // rejects the exact 2^64 boundary that the looser `>` check missed.
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint32(0))+1 {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(uint32(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(uint32(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -1406,6 +1531,7 @@ func (b *Uint32Builder) Unmarshal(dec *json.Decoder) error {
func (b *Uint32Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -1647,6 +1773,7 @@ func (b *Float32Builder) Unmarshal(dec *json.Decoder) error {
func (b *Float32Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -1844,10 +1971,11 @@ func (b *Int16Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any int
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseInt(v, 10, 2*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -1856,18 +1984,25 @@ func (b *Int16Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
minVal := float64(int64(-1) << (2*8 - 1))
maxVal := float64(int64(1) << (2*8 - 1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(int16(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := int64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(int16(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -1880,10 +2015,13 @@ func (b *Int16Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(int16(v))
case json.Number:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON like 1.5 or 128 (for int8) is surfaced as
+ // UnmarshalTypeError rather than silently truncated or wrapped.
f, err := strconv.ParseInt(v.String(), 10, 2*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -1892,18 +2030,25 @@ func (b *Int16Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
minVal := float64(int64(-1) << (2*8 - 1))
maxVal := float64(int64(1) << (2*8 - 1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(int16(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := int64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(int16(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -1936,6 +2081,7 @@ func (b *Int16Builder) Unmarshal(dec *json.Decoder) error {
func (b *Int16Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -2133,10 +2279,11 @@ func (b *Uint16Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any uint
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v, 10, 2*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -2145,16 +2292,23 @@ func (b *Uint16Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint16(0)) {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint16(0))+1 {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(uint16(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(uint16(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -2167,10 +2321,14 @@ func (b *Uint16Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(uint16(v))
case json.Number:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON is surfaced as UnmarshalTypeError rather than silently
+ // coerced. Range-check before any uint conversion to avoid undefined
+ // behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v.String(), 10, 2*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -2179,16 +2337,26 @@ func (b *Uint16Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint16(0)) {
+ // Boundary is max+1: for uint16, valid range is [0, max]; reject anything >= max+1.
+ // For uint64, float64(^uint64(0)) already rounds up to 2^64, so this naturally
+ // rejects the exact 2^64 boundary that the looser `>` check missed.
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint16(0))+1 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(uint16(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(uint16(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -2221,6 +2389,7 @@ func (b *Uint16Builder) Unmarshal(dec *json.Decoder) error {
func (b *Uint16Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -2418,10 +2587,11 @@ func (b *Int8Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any int
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseInt(v, 10, 1*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -2430,18 +2600,25 @@ func (b *Int8Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
minVal := float64(int64(-1) << (1*8 - 1))
maxVal := float64(int64(1) << (1*8 - 1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(int8(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := int64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(int8(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -2454,10 +2631,13 @@ func (b *Int8Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(int8(v))
case json.Number:
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON like 1.5 or 128 (for int8) is surfaced as
+ // UnmarshalTypeError rather than silently truncated or wrapped.
f, err := strconv.ParseInt(v.String(), 10, 1*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -2466,18 +2646,25 @@ func (b *Int8Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
minVal := float64(int64(-1) << (1*8 - 1))
maxVal := float64(int64(1) << (1*8 - 1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(int8(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ truncated := int64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(int8(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -2510,6 +2697,7 @@ func (b *Int8Builder) Unmarshal(dec *json.Decoder) error {
func (b *Int8Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -2707,10 +2895,11 @@ func (b *Uint8Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
case string:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any uint
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v, 10, 1*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -2719,16 +2908,23 @@ func (b *Uint8Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint8(0)) {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint8(0))+1 {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf(uint8(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf(uint8(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -2741,10 +2937,14 @@ func (b *Uint8Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(uint8(v))
case json.Number:
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON is surfaced as UnmarshalTypeError rather than silently
+ // coerced. Range-check before any uint conversion to avoid undefined
+ // behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v.String(), 10, 1*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -2753,16 +2953,26 @@ func (b *Uint8Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^uint8(0)) {
+ // Boundary is max+1: for uint8, valid range is [0, max]; reject anything >= max+1.
+ // For uint64, float64(^uint64(0)) already rounds up to 2^64, so this naturally
+ // rejects the exact 2^64 boundary that the looser `>` check missed.
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^uint8(0))+1 {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf(uint8(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(uint8(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
if err != nil {
return &json.UnmarshalTypeError{
@@ -2795,6 +3005,7 @@ func (b *Uint8Builder) Unmarshal(dec *json.Decoder) error {
func (b *Uint8Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -2973,6 +3184,10 @@ func (b *Time32Builder) AppendValueFromString(s string) error {
b.AppendNull()
return nil
}
+ if v, parseErr := strconv.ParseInt(s, 10, 32); parseErr == nil {
+ b.Append(arrow.Time32(v))
+ return nil
+ }
val, err := arrow.Time32FromString(s, b.dtype.Unit)
if err != nil {
b.AppendNull()
@@ -2992,6 +3207,10 @@ func (b *Time32Builder) UnmarshalOne(dec *json.Decoder) error {
case nil:
b.AppendNull()
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, 4*8); parseErr == nil {
+ b.Append(arrow.Time32(i))
+ break
+ }
tm, err := arrow.Time32FromString(v, b.dtype.Unit)
if err != nil {
return &json.UnmarshalTypeError{
@@ -3011,6 +3230,15 @@ func (b *Time32Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
+ // arrow.Time32 is int32-backed; reject values outside int32 range
+ // to avoid silent wrap-around.
+ if n < -2147483648 || n > 2147483647 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(arrow.Time32(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
b.Append(arrow.Time32(n))
case float64:
b.Append(arrow.Time32(v))
@@ -3037,6 +3265,7 @@ func (b *Time32Builder) Unmarshal(dec *json.Decoder) error {
func (b *Time32Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -3215,6 +3444,10 @@ func (b *Time64Builder) AppendValueFromString(s string) error {
b.AppendNull()
return nil
}
+ if v, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Time64(v))
+ return nil
+ }
val, err := arrow.Time64FromString(s, b.dtype.Unit)
if err != nil {
b.AppendNull()
@@ -3234,6 +3467,10 @@ func (b *Time64Builder) UnmarshalOne(dec *json.Decoder) error {
case nil:
b.AppendNull()
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, 8*8); parseErr == nil {
+ b.Append(arrow.Time64(i))
+ break
+ }
tm, err := arrow.Time64FromString(v, b.dtype.Unit)
if err != nil {
return &json.UnmarshalTypeError{
@@ -3279,6 +3516,7 @@ func (b *Time64Builder) Unmarshal(dec *json.Decoder) error {
func (b *Time64Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -3456,6 +3694,10 @@ func (b *Date32Builder) AppendValueFromString(s string) error {
b.AppendNull()
return nil
}
+ if v, parseErr := strconv.ParseInt(s, 10, 32); parseErr == nil {
+ b.Append(arrow.Date32(v))
+ return nil
+ }
tm, err := time.Parse("2006-01-02", s)
if err != nil {
b.AppendNull()
@@ -3475,6 +3717,10 @@ func (b *Date32Builder) UnmarshalOne(dec *json.Decoder) error {
case nil:
b.AppendNull()
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, 4*8); parseErr == nil {
+ b.Append(arrow.Date32(i))
+ break
+ }
tm, err := time.Parse("2006-01-02", v)
if err != nil {
return &json.UnmarshalTypeError{
@@ -3494,6 +3740,15 @@ func (b *Date32Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
+ // arrow.Date32 is int32-backed; reject values outside int32 range
+ // to avoid silent wrap-around.
+ if n < -2147483648 || n > 2147483647 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf(arrow.Date32(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
b.Append(arrow.Date32(n))
case float64:
b.Append(arrow.Date32(v))
@@ -3520,6 +3775,7 @@ func (b *Date32Builder) Unmarshal(dec *json.Decoder) error {
func (b *Date32Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -3697,6 +3953,10 @@ func (b *Date64Builder) AppendValueFromString(s string) error {
b.AppendNull()
return nil
}
+ if v, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Date64(v))
+ return nil
+ }
tm, err := time.Parse("2006-01-02", s)
if err != nil {
b.AppendNull()
@@ -3716,6 +3976,10 @@ func (b *Date64Builder) UnmarshalOne(dec *json.Decoder) error {
case nil:
b.AppendNull()
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, 8*8); parseErr == nil {
+ b.Append(arrow.Date64(i))
+ break
+ }
tm, err := time.Parse("2006-01-02", v)
if err != nil {
return &json.UnmarshalTypeError{
@@ -3761,6 +4025,7 @@ func (b *Date64Builder) Unmarshal(dec *json.Decoder) error {
func (b *Date64Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -3939,6 +4204,10 @@ func (b *DurationBuilder) AppendValueFromString(s string) error {
b.AppendNull()
return nil
}
+ if v, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Duration(v))
+ return nil
+ }
dur, err := time.ParseDuration(s)
if err != nil {
return err
@@ -3970,6 +4239,13 @@ func (b *DurationBuilder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append(arrow.Duration(v))
case string:
+ // raw integer strings (e.g. "9223372036854775807") - useful when the
+ // caller serializes large integers as strings to bypass JSON number
+ // precision limits
+ if i, parseErr := strconv.ParseInt(v, 10, 64); parseErr == nil {
+ b.Append(arrow.Duration(i))
+ break
+ }
// be flexible for specifying durations by accepting forms like
// 3h2m0.5s regardless of the unit and converting it to the proper
// precision.
@@ -4026,6 +4302,7 @@ func (b *DurationBuilder) Unmarshal(dec *json.Decoder) error {
func (b *DurationBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/numericbuilder.gen.go.tmpl b/arrow/array/numericbuilder.gen.go.tmpl
index 5569b2eb..1f7d433e 100644
--- a/arrow/array/numericbuilder.gen.go.tmpl
+++ b/arrow/array/numericbuilder.gen.go.tmpl
@@ -17,6 +17,7 @@
package array
import (
+ "math"
"reflect"
"strconv"
@@ -213,6 +214,10 @@ func (b *{{.Name}}Builder) AppendValueFromString(s string) error {
return nil
}
{{if or (eq .Name "Date32") -}}
+ if v, parseErr := strconv.ParseInt(s, 10, 32); parseErr == nil {
+ b.Append(arrow.Date32(v))
+ return nil
+ }
tm, err := time.Parse("2006-01-02", s)
if err != nil {
b.AppendNull()
@@ -220,6 +225,10 @@ func (b *{{.Name}}Builder) AppendValueFromString(s string) error {
}
b.Append(arrow.Date32FromTime(tm))
{{else if or (eq .Name "Date64") -}}
+ if v, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Date64(v))
+ return nil
+ }
tm, err := time.Parse("2006-01-02", s)
if err != nil {
b.AppendNull()
@@ -227,6 +236,10 @@ func (b *{{.Name}}Builder) AppendValueFromString(s string) error {
}
b.Append(arrow.Date64FromTime(tm))
{{else if or (eq .Name "Time32") -}}
+ if v, parseErr := strconv.ParseInt(s, 10, 32); parseErr == nil {
+ b.Append(arrow.Time32(v))
+ return nil
+ }
val, err := arrow.Time32FromString(s, b.dtype.Unit)
if err != nil {
b.AppendNull()
@@ -234,6 +247,10 @@ func (b *{{.Name}}Builder) AppendValueFromString(s string) error {
}
b.Append(val)
{{else if or (eq .Name "Time64") -}}
+ if v, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Time64(v))
+ return nil
+ }
val, err := arrow.Time64FromString(s, b.dtype.Unit)
if err != nil {
b.AppendNull()
@@ -241,6 +258,10 @@ func (b *{{.Name}}Builder) AppendValueFromString(s string) error {
}
b.Append(val)
{{else if (eq .Name "Duration") -}}
+ if v, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Duration(v))
+ return nil
+ }
dur, err := time.ParseDuration(s)
if err != nil {
return err
@@ -283,6 +304,10 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
b.AppendNull()
{{if or (eq .Name "Date32") (eq .Name "Date64") -}}
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, {{.Size}}*8); parseErr == nil {
+ b.Append({{.QualifiedType}}(i))
+ break
+ }
tm, err := time.Parse("2006-01-02", v)
if err != nil {
return &json.UnmarshalTypeError{
@@ -302,11 +327,26 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
+ {{if eq .Size "4" -}}
+ // {{.QualifiedType}} is int32-backed; reject values outside int32 range
+ // to avoid silent wrap-around.
+ if n < -2147483648 || n > 2147483647 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf({{.QualifiedType}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{end -}}
b.Append({{.QualifiedType}}(n))
case float64:
b.Append({{.QualifiedType}}(v))
{{else if or (eq .Name "Time32") (eq .Name "Time64") -}}
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, {{.Size}}*8); parseErr == nil {
+ b.Append({{.QualifiedType}}(i))
+ break
+ }
tm, err := {{.QualifiedType}}FromString(v, b.dtype.Unit)
if err != nil {
return &json.UnmarshalTypeError{
@@ -326,6 +366,17 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
+ {{if eq .Size "4" -}}
+ // {{.QualifiedType}} is int32-backed; reject values outside int32 range
+ // to avoid silent wrap-around.
+ if n < -2147483648 || n > 2147483647 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf({{.QualifiedType}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{end -}}
b.Append({{.QualifiedType}}(n))
case float64:
b.Append({{.QualifiedType}}(v))
@@ -343,6 +394,13 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
case float64:
b.Append({{.QualifiedType}}(v))
case string:
+ // raw integer strings (e.g. "9223372036854775807") - useful when the
+ // caller serializes large integers as strings to bypass JSON number
+ // precision limits
+ if i, parseErr := strconv.ParseInt(v, 10, 64); parseErr == nil {
+ b.Append(arrow.Duration(i))
+ break
+ }
// be flexible for specifying durations by accepting forms like
// 3h2m0.5s regardless of the unit and converting it to the proper
// precision.
@@ -381,10 +439,11 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
{{if or (eq .Name "Float32") (eq .Name "Float64") -}}
f, err := strconv.ParseFloat(v, {{.Size}}*8)
{{else if eq (printf "%.1s" .Name) "U" -}}
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any uint
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v, 10, {{.Size}}*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -393,22 +452,42 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^{{.name}}(0)) {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^{{.name}}(0))+1 {
return &json.UnmarshalTypeError{
Value: v,
Type: reflect.TypeOf({{.name}}(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ {{if eq .Size "8" -}}
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{end -}}
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
{{else -}}
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first for direct integer strings, fall back to ParseFloat for
+ // exponential notation. Reject NaN/Inf, then range-check before any int
+ // conversion to avoid undefined behavior on overflow or non-finite values.
f, err := strconv.ParseInt(v, 10, {{.Size}}*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v, 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -417,13 +496,12 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
{{if eq .Size "8" -}}
- if fval != float64(int64(fval)) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
{{else -}}
minVal := float64(int64(-1) << ({{.Size}}*8-1))
maxVal := float64(int64(1) << ({{.Size}}*8-1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
{{end -}}
return &json.UnmarshalTypeError{
Value: v,
@@ -431,8 +509,28 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ {{if eq .Size "8" -}}
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{end -}}
+ truncated := int64(fval)
+ if fval != float64(truncated) {
+ return &json.UnmarshalTypeError{
+ Value: v,
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ f = truncated
+ err = nil
}
{{end -}}
if err != nil {
@@ -449,10 +547,14 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
{{if or (eq .Name "Float32") (eq .Name "Float64") -}}
f, err := strconv.ParseFloat(v.String(), {{.Size}}*8)
{{else if eq (printf "%.1s" .Name) "U" -}}
- // Try ParseUint first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseUint first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON is surfaced as UnmarshalTypeError rather than silently
+ // coerced. Range-check before any uint conversion to avoid undefined
+ // behavior on overflow or non-finite values.
f, err := strconv.ParseUint(v.String(), 10, {{.Size}}*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -461,22 +563,47 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
- if fval != float64(uint64(fval)) || fval < 0 || fval > float64(^{{.name}}(0)) {
+ // Boundary is max+1: for {{.name}}, valid range is [0, max]; reject anything >= max+1.
+ // For uint64, float64(^uint64(0)) already rounds up to 2^64, so this naturally
+ // rejects the exact 2^64 boundary that the looser `>` check missed.
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < 0 || fval >= float64(^{{.name}}(0))+1 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{if eq .Size "8" -}}
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{end -}}
+ truncated := uint64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf({{.name}}(0)),
Offset: dec.InputOffset(),
}
}
- f = uint64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
{{else -}}
- // Try ParseInt first for direct integer strings, fall back to ParseFloat for exponential notation
+ // Try ParseInt first to preserve precision for integer values too large
+ // for float64. Fall back to ParseFloat to support exponential notation
+ // (e.g. "1e3"), but reject NaN/Inf, fractional, or out-of-range values
+ // so invalid JSON like 1.5 or 128 (for int8) is surfaced as
+ // UnmarshalTypeError rather than silently truncated or wrapped.
f, err := strconv.ParseInt(v.String(), 10, {{.Size}}*8)
if err != nil {
- // Could be exponential notation - try parsing as float
fval, ferr := strconv.ParseFloat(v.String(), 64)
if ferr != nil {
return &json.UnmarshalTypeError{
@@ -485,22 +612,41 @@ func (b *{{.Name}}Builder) UnmarshalOne(dec *json.Decoder) error {
Offset: dec.InputOffset(),
}
}
- // Check if it's a whole number and in valid range
{{if eq .Size "8" -}}
- if fval != float64(int64(fval)) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < -9223372036854775808.0 || fval >= 9223372036854775808.0 {
{{else -}}
minVal := float64(int64(-1) << ({{.Size}}*8-1))
maxVal := float64(int64(1) << ({{.Size}}*8-1))
- if fval != float64(int64(fval)) || fval < minVal || fval >= maxVal {
+ if math.IsNaN(fval) || math.IsInf(fval, 0) || fval < minVal || fval >= maxVal {
+ {{end -}}
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
+ {{if eq .Size "8" -}}
+ // Beyond 2^53, float64 cannot represent every integer exactly, so
+ // exponent-form input like "9.007199254740993e15" may have been
+ // silently rounded by ParseFloat. Reject conservatively.
+ if math.Abs(fval) >= 9007199254740992 {
+ return &json.UnmarshalTypeError{
+ Value: v.String(),
+ Type: reflect.TypeOf({{.name}}(0)),
+ Offset: dec.InputOffset(),
+ }
+ }
{{end -}}
+ truncated := int64(fval)
+ if fval != float64(truncated) {
return &json.UnmarshalTypeError{
Value: v.String(),
Type: reflect.TypeOf({{.name}}(0)),
Offset: dec.InputOffset(),
}
}
- f = int64(fval)
- err = nil // Clear error after successful float parsing
+ f = truncated
+ err = nil
}
{{end -}}
if err != nil {
@@ -534,6 +680,7 @@ func (b *{{.Name}}Builder) Unmarshal(dec *json.Decoder) error {
func (b *{{.Name}}Builder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/numericbuilder.gen_test.go b/arrow/array/numericbuilder.gen_test.go
index 1336815b..2ca56329 100644
--- a/arrow/array/numericbuilder.gen_test.go
+++ b/arrow/array/numericbuilder.gen_test.go
@@ -237,7 +237,7 @@ func TestInt64BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewInt64Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -251,6 +251,7 @@ func TestInt64BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -463,7 +464,7 @@ func TestUint64BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewUint64Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -477,6 +478,7 @@ func TestUint64BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -689,7 +691,7 @@ func TestFloat64BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewFloat64Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, "+Inf", 2, 3, "NaN", "NaN", 4, 5, "-Inf"]`
+ jsonstr := `[0, 1, "+Inf", 2.5, 3, "NaN", "NaN", 4, 5, "-Inf"]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -702,6 +704,7 @@ func TestFloat64BuilderUnmarshalJSON(t *testing.T) {
assert.False(t, math.IsInf(float64(arr.Value(0)), 0), arr.Value(0))
assert.True(t, math.IsInf(float64(arr.Value(2)), 1), arr.Value(2))
assert.True(t, math.IsNaN(float64(arr.Value(5))), arr.Value(5))
+ assert.Equal(t, float64(2.5), arr.Value(3))
}
func TestInt32StringRoundTrip(t *testing.T) {
@@ -913,7 +916,7 @@ func TestInt32BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewInt32Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -927,6 +930,7 @@ func TestInt32BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -1139,7 +1143,7 @@ func TestUint32BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewUint32Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -1153,6 +1157,7 @@ func TestUint32BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -1365,7 +1370,7 @@ func TestFloat32BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewFloat32Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, "+Inf", 2, 3, "NaN", "NaN", 4, 5, "-Inf"]`
+ jsonstr := `[0, 1, "+Inf", 2.5, 3, "NaN", "NaN", 4, 5, "-Inf"]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -1378,6 +1383,7 @@ func TestFloat32BuilderUnmarshalJSON(t *testing.T) {
assert.False(t, math.IsInf(float64(arr.Value(0)), 0), arr.Value(0))
assert.True(t, math.IsInf(float64(arr.Value(2)), 1), arr.Value(2))
assert.True(t, math.IsNaN(float64(arr.Value(5))), arr.Value(5))
+ assert.Equal(t, float32(2.5), arr.Value(3))
}
func TestInt16StringRoundTrip(t *testing.T) {
@@ -1589,7 +1595,7 @@ func TestInt16BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewInt16Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -1603,6 +1609,7 @@ func TestInt16BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -1815,7 +1822,7 @@ func TestUint16BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewUint16Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -1829,6 +1836,7 @@ func TestUint16BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -2041,7 +2049,7 @@ func TestInt8BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewInt8Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -2055,6 +2063,7 @@ func TestInt8BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -2267,7 +2276,7 @@ func TestUint8BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewUint8Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -2281,6 +2290,7 @@ func TestUint8BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -2499,7 +2509,7 @@ func TestTime32BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewTime32Builder(mem, dtype)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -2513,6 +2523,7 @@ func TestTime32BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -2731,7 +2742,7 @@ func TestTime64BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewTime64Builder(mem, dtype)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -2745,6 +2756,7 @@ func TestTime64BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -2957,7 +2969,7 @@ func TestDate32BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewDate32Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -2971,6 +2983,7 @@ func TestDate32BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -3190,7 +3203,7 @@ func TestDate64BuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewDate64Builder(mem)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -3204,6 +3217,7 @@ func TestDate64BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
@@ -3422,7 +3436,7 @@ func TestDurationBuilderUnmarshalJSON(t *testing.T) {
bldr := array.NewDurationBuilder(mem, dtype)
defer bldr.Release()
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -3436,5 +3450,6 @@ func TestDurationBuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
}
diff --git a/arrow/array/numericbuilder.gen_test.go.tmpl b/arrow/array/numericbuilder.gen_test.go.tmpl
index 86cc74a5..07d5fc8a 100644
--- a/arrow/array/numericbuilder.gen_test.go.tmpl
+++ b/arrow/array/numericbuilder.gen_test.go.tmpl
@@ -287,7 +287,7 @@ func Test{{.Name}}BuilderUnmarshalJSON(t *testing.T) {
{{ if or (eq .Name "Float64") (eq .Name "Float32") -}}
- jsonstr := `[0, 1, "+Inf", 2, 3, "NaN", "NaN", 4, 5, "-Inf"]`
+ jsonstr := `[0, 1, "+Inf", 2.5, 3, "NaN", "NaN", 4, 5, "-Inf"]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -300,8 +300,9 @@ func Test{{.Name}}BuilderUnmarshalJSON(t *testing.T) {
assert.False(t, math.IsInf(float64(arr.Value(0)), 0), arr.Value(0))
assert.True(t, math.IsInf(float64(arr.Value(2)), 1), arr.Value(2))
assert.True(t, math.IsNaN(float64(arr.Value(5))), arr.Value(5))
+ assert.Equal(t, {{ .Type }}(2.5), arr.Value(3))
{{else}}
- jsonstr := `[0, 1, null, 2.3, -11]`
+ jsonstr := `[0, 1, null, 2, 100]`
err := bldr.UnmarshalJSON([]byte(jsonstr))
assert.NoError(t, err)
@@ -315,6 +316,7 @@ func Test{{.Name}}BuilderUnmarshalJSON(t *testing.T) {
assert.Equal(t, int64(1), int64(arr.Value(1)))
assert.True(t, arr.IsNull(2))
assert.Equal(t, int64(2), int64(arr.Value(3)))
+ assert.Equal(t, int64(100), int64(arr.Value(4)))
assert.Equal(t, int64(5), int64(arr.Len()))
{{end -}}
}
diff --git a/arrow/array/record.go b/arrow/array/record.go
index 0aaf771b..17bfb783 100644
--- a/arrow/array/record.go
+++ b/arrow/array/record.go
@@ -384,12 +384,15 @@ func (b *RecordBuilder) NewRecord() arrow.Record {
return b.NewRecordBatch()
}
-// UnmarshalJSON for record builder will read in a single object and add the values
-// to each field in the recordbuilder, missing fields will get a null and unexpected
-// keys will be ignored. If reading in an array of records as a single batch, then use
-// a structbuilder and use RecordFromStruct.
-func (b *RecordBuilder) UnmarshalJSON(data []byte) error {
- dec := json.NewDecoder(bytes.NewReader(data))
+// UnmarshalOne reads one row (a JSON object) from the supplied decoder and
+// appends a value to each field in the RecordBuilder. Missing fields are
+// appended as nulls and unrecognized keys are silently ignored.
+//
+// Unlike UnmarshalJSON, this method receives an already-configured
+// json.Decoder, so options such as UseNumber set by the caller are honored
+// for nested field decoding. This is critical for preserving large integer
+// values (>2^53) that cannot be represented exactly as float64.
+func (b *RecordBuilder) UnmarshalOne(dec *json.Decoder) error {
// should start with a '{'
t, err := dec.Token()
if err != nil {
@@ -427,6 +430,11 @@ func (b *RecordBuilder) UnmarshalJSON(data []byte) error {
}
}
+ // consume the closing '}'
+ if _, err := dec.Token(); err != nil {
+ return err
+ }
+
for i := 0; i < b.schema.NumFields(); i++ {
if !keylist[b.schema.Field(i).Name] {
b.fields[i].AppendNull()
@@ -435,6 +443,33 @@ func (b *RecordBuilder) UnmarshalJSON(data []byte) error {
return nil
}
+// Unmarshal reads multiple rows from the decoder, calling UnmarshalOne in a
+// loop until dec.More() reports there are no more values. Like UnmarshalOne,
+// this honors decoder configuration such as UseNumber set by the caller.
+func (b *RecordBuilder) Unmarshal(dec *json.Decoder) error {
+ for dec.More() {
+ if err := b.UnmarshalOne(dec); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// UnmarshalJSON for record builder will read in a single object and add the values
+// to each field in the recordbuilder, missing fields will get a null and unexpected
+// keys will be ignored. If reading in an array of records as a single batch, then use
+// a structbuilder and use RecordFromStruct.
+//
+// UseNumber is enabled on the internal decoder so that integer values too large
+// to be represented exactly as float64 (e.g. values beyond 2^53) are preserved.
+// Callers who need full control over decoder configuration should use
+// UnmarshalOne with a pre-configured json.Decoder instead.
+func (b *RecordBuilder) UnmarshalJSON(data []byte) error {
+ dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
+ return b.UnmarshalOne(dec)
+}
+
type iterReader struct {
refCount atomic.Int64
diff --git a/arrow/array/string.go b/arrow/array/string.go
index 60323e37..35b412f2 100644
--- a/arrow/array/string.go
+++ b/arrow/array/string.go
@@ -629,6 +629,7 @@ func (b *StringBuilder) Unmarshal(dec *json.Decoder) error {
func (b *StringBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -722,6 +723,7 @@ func (b *LargeStringBuilder) Unmarshal(dec *json.Decoder) error {
func (b *LargeStringBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -788,6 +790,7 @@ func (b *StringViewBuilder) Unmarshal(dec *json.Decoder) error {
func (b *StringViewBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/struct.go b/arrow/array/struct.go
index 6883712c..7df6d8d4 100644
--- a/arrow/array/struct.go
+++ b/arrow/array/struct.go
@@ -520,6 +520,7 @@ func (b *StructBuilder) Unmarshal(dec *json.Decoder) error {
func (b *StructBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/timestamp.go b/arrow/array/timestamp.go
index d0b5b062..5ac0ee6c 100644
--- a/arrow/array/timestamp.go
+++ b/arrow/array/timestamp.go
@@ -20,6 +20,7 @@ import (
"bytes"
"fmt"
"reflect"
+ "strconv"
"strings"
"time"
@@ -315,6 +316,11 @@ func (b *TimestampBuilder) AppendValueFromString(s string) error {
return nil
}
+ if i, parseErr := strconv.ParseInt(s, 10, 64); parseErr == nil {
+ b.Append(arrow.Timestamp(i))
+ return nil
+ }
+
loc, err := b.dtype.GetZone()
if err != nil {
return err
@@ -339,6 +345,10 @@ func (b *TimestampBuilder) UnmarshalOne(dec *json.Decoder) error {
case nil:
b.AppendNull()
case string:
+ if i, parseErr := strconv.ParseInt(v, 10, 64); parseErr == nil {
+ b.Append(arrow.Timestamp(i))
+ break
+ }
loc, _ := b.dtype.GetZone()
tm, _, err := arrow.TimestampFromStringInLocation(v, b.dtype.Unit, loc)
if err != nil {
@@ -385,6 +395,7 @@ func (b *TimestampBuilder) Unmarshal(dec *json.Decoder) error {
func (b *TimestampBuilder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/union.go b/arrow/array/union.go
index 9c13af05..beaaf4e1 100644
--- a/arrow/array/union.go
+++ b/arrow/array/union.go
@@ -998,6 +998,7 @@ func (b *SparseUnionBuilder) NewSparseUnionArray() (a *SparseUnion) {
func (b *SparseUnionBuilder) UnmarshalJSON(data []byte) (err error) {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
@@ -1257,6 +1258,7 @@ func (b *DenseUnionBuilder) NewDenseUnionArray() (a *DenseUnion) {
func (b *DenseUnionBuilder) UnmarshalJSON(data []byte) (err error) {
dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
t, err := dec.Token()
if err != nil {
return err
diff --git a/arrow/array/util.go b/arrow/array/util.go
index 5cbcbeab..136ed353 100644
--- a/arrow/array/util.go
+++ b/arrow/array/util.go
@@ -61,10 +61,13 @@ func WithStartOffset(off int64) FromJSONOption {
}
}
-// WithUseNumber enables the 'UseNumber' option on the json decoder, using
-// the json.Number type instead of assuming float64 for numbers. This is critical
-// if you have numbers that are larger than what can fit into the 53 bits of
-// an IEEE float64 mantissa and want to preserve its value.
+// WithUseNumber previously enabled the 'UseNumber' option on the json decoder.
+// As of issue #804, FromJSON, RecordFromJSON, and TableFromJSON unconditionally
+// enable UseNumber so that integer values too large to fit in float64 (i.e.
+// beyond 2^53) are preserved without silent corruption. This option is now a
+// no-op and is retained for backward compatibility.
+//
+// Deprecated: UseNumber is now always enabled; this option has no effect.
func WithUseNumber() FromJSONOption {
return func(c *fromJSONCfg) {
c.useNumber = true
@@ -143,16 +146,13 @@ func FromJSON(mem memory.Allocator, dt arrow.DataType, r io.Reader, opts ...From
defer bldr.Release()
dec := json.NewDecoder(r)
+ dec.UseNumber()
defer func() {
if errors.Is(err, io.EOF) {
err = fmt.Errorf("failed parsing json: %w", io.ErrUnexpectedEOF)
}
}()
- if cfg.useNumber {
- dec.UseNumber()
- }
-
if !cfg.multiDocument {
t, err := dec.Token()
if err != nil {
@@ -237,9 +237,7 @@ func RecordFromJSON(mem memory.Allocator, schema *arrow.Schema, r io.Reader, opt
defer bldr.Release()
dec := json.NewDecoder(r)
- if cfg.useNumber {
- dec.UseNumber()
- }
+ dec.UseNumber()
if !cfg.multiDocument {
t, err := dec.Token()
@@ -251,7 +249,7 @@ func RecordFromJSON(mem memory.Allocator, schema *arrow.Schema, r io.Reader, opt
}
for dec.More() {
- if err := dec.Decode(bldr); err != nil {
+ if err := bldr.UnmarshalOne(dec); err != nil {
return nil, dec.InputOffset(), fmt.Errorf("failed to decode json: %w", err)
}
}
@@ -265,8 +263,7 @@ func RecordFromJSON(mem memory.Allocator, schema *arrow.Schema, r io.Reader, opt
}
for {
- err := dec.Decode(bldr)
- if err != nil {
+ if err := bldr.UnmarshalOne(dec); err != nil {
if errors.Is(err, io.EOF) {
break
}
diff --git a/arrow/array/util_test.go b/arrow/array/util_test.go
index fb837871..eb3de6a8 100644
--- a/arrow/array/util_test.go
+++ b/arrow/array/util_test.go
@@ -543,3 +543,264 @@ func TestRecordBuilderUnmarshalJSONExtraFields(t *testing.T) {
assert.Truef(t, array.RecordEqual(rec1, rec2), "expected: %s\nactual: %s", rec1, rec2)
}
+
+func TestRecordFromJSONLargeInt64Default(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ schema := arrow.NewSchema([]arrow.Field{
+ {Name: "a", Type: arrow.PrimitiveTypes.Int64},
+ }, nil)
+
+ const data = `[{"a": 9223372036854775807}, {"a": -9223372036854775808}]`
+ batch, _, err := array.RecordFromJSON(mem, schema, strings.NewReader(data))
+ require.NoError(t, err)
+ require.NotNil(t, batch)
+ defer batch.Release()
+
+ col := batch.Column(0).(*array.Int64)
+ assert.EqualValues(t, int64(9223372036854775807), col.Value(0))
+ assert.EqualValues(t, int64(-9223372036854775808), col.Value(1))
+}
+
+func TestRecordFromJSONLargeInt64WithUseNumber(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ schema := arrow.NewSchema([]arrow.Field{
+ {Name: "a", Type: arrow.PrimitiveTypes.Int64},
+ }, nil)
+
+ const data = `[{"a": 9223372036854775807}, {"a": -9223372036854775808}]`
+ //nolint:staticcheck // SA1019: explicitly verifying deprecated WithUseNumber still works
+ batch, _, err := array.RecordFromJSON(mem, schema, strings.NewReader(data), array.WithUseNumber())
+ require.NoError(t, err)
+ require.NotNil(t, batch)
+ defer batch.Release()
+
+ col := batch.Column(0).(*array.Int64)
+ assert.EqualValues(t, int64(9223372036854775807), col.Value(0))
+ assert.EqualValues(t, int64(-9223372036854775808), col.Value(1))
+}
+
+func TestRecordFromJSONLargeDuration(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ schema := arrow.NewSchema([]arrow.Field{
+ {Name: "a", Type: arrow.FixedWidthTypes.Duration_s},
+ }, nil)
+
+ const data = `[{"a": 9223372036854775807}]`
+ batch, _, err := array.RecordFromJSON(mem, schema, strings.NewReader(data))
+ require.NoError(t, err)
+ require.NotNil(t, batch)
+ defer batch.Release()
+
+ col := batch.Column(0).(*array.Duration)
+ assert.EqualValues(t, arrow.Duration(9223372036854775807), col.Value(0))
+}
+
+func TestRecordBuilderUnmarshalJSONLargeInt64(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ schema := arrow.NewSchema([]arrow.Field{
+ {Name: "a", Type: arrow.PrimitiveTypes.Int64},
+ }, nil)
+
+ bldr := array.NewRecordBuilder(mem, schema)
+ defer bldr.Release()
+
+ require.NoError(t, bldr.UnmarshalJSON([]byte(`{"a": 9223372036854775807}`)))
+
+ rec := bldr.NewRecordBatch()
+ defer rec.Release()
+
+ col := rec.Column(0).(*array.Int64)
+ assert.EqualValues(t, int64(9223372036854775807), col.Value(0))
+}
+
+func TestRecordBuilderUnmarshalOnePreservesUserDecoderOptions(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ schema := arrow.NewSchema([]arrow.Field{
+ {Name: "a", Type: arrow.PrimitiveTypes.Int64},
+ }, nil)
+
+ bldr := array.NewRecordBuilder(mem, schema)
+ defer bldr.Release()
+
+ src := strings.NewReader(`{"a": 9223372036854775807}`)
+ dec := json.NewDecoder(src)
+ dec.UseNumber()
+
+ require.NoError(t, bldr.UnmarshalOne(dec))
+
+ rec := bldr.NewRecordBatch()
+ defer rec.Release()
+
+ col := rec.Column(0).(*array.Int64)
+ assert.EqualValues(t, int64(9223372036854775807), col.Value(0))
+}
+
+func TestDurationBuilderJSONStringInteger(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Duration_s,
+ strings.NewReader(`["9223372036854775807"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Duration)
+ assert.EqualValues(t, arrow.Duration(9223372036854775807), col.Value(0))
+}
+
+func TestTimestampBuilderJSONStringInteger(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Timestamp_s,
+ strings.NewReader(`["9223372036854775807"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Timestamp)
+ assert.EqualValues(t, arrow.Timestamp(9223372036854775807), col.Value(0))
+}
+
+func TestTime32BuilderJSONStringInteger(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Time32s,
+ strings.NewReader(`["2147483647"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Time32)
+ assert.EqualValues(t, arrow.Time32(2147483647), col.Value(0))
+}
+
+func TestTime64BuilderJSONStringInteger(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Time64us,
+ strings.NewReader(`["9223372036854775807"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Time64)
+ assert.EqualValues(t, arrow.Time64(9223372036854775807), col.Value(0))
+}
+
+func TestDate32BuilderJSONStringInteger(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Date32,
+ strings.NewReader(`["2147483647"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Date32)
+ assert.EqualValues(t, arrow.Date32(2147483647), col.Value(0))
+}
+
+func TestDate64BuilderJSONStringInteger(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Date64,
+ strings.NewReader(`["9223372036854775807"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Date64)
+ assert.EqualValues(t, arrow.Date64(9223372036854775807), col.Value(0))
+}
+
+func TestDurationBuilderJSONStringIntegerInvalid(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ _, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Duration_s,
+ strings.NewReader(`["abc"]`))
+ assert.Error(t, err)
+}
+
+func TestDurationBuilderJSONStringDurationFormat(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Duration_s,
+ strings.NewReader(`["3h2m0.5s"]`))
+ require.NoError(t, err)
+ require.NotNil(t, arr)
+ defer arr.Release()
+
+ col := arr.(*array.Duration)
+ assert.EqualValues(t, arrow.Duration(10920), col.Value(0))
+}
+
+// TestJSONNumberStrictValidation verifies that with UseNumber always enabled
+// (issue #804), invalid integer JSON inputs are rejected rather than silently
+// truncated, wrapped, or coerced.
+func TestJSONNumberStrictValidation(t *testing.T) {
+ cases := []struct {
+ name string
+ dt arrow.DataType
+ json string
+ }{
+ {"Int64Fractional", arrow.PrimitiveTypes.Int64, `[1.5]`},
+ {"Int8OutOfRangePositive", arrow.PrimitiveTypes.Int8, `[128]`},
+ {"Int8OutOfRangeNegative", arrow.PrimitiveTypes.Int8, `[-129]`},
+ {"Int16OutOfRange", arrow.PrimitiveTypes.Int16, `[32768]`},
+ {"Uint8Negative", arrow.PrimitiveTypes.Uint8, `[-1]`},
+ {"Uint8Fractional", arrow.PrimitiveTypes.Uint8, `[0.5]`},
+ {"Uint16OutOfRange", arrow.PrimitiveTypes.Uint16, `[65536]`},
+ {"Uint64Negative", arrow.PrimitiveTypes.Uint64, `[-1]`},
+ {"Uint64ExactBoundary", arrow.PrimitiveTypes.Uint64, `[18446744073709551616]`},
+ {"Uint64ExponentialOverflow", arrow.PrimitiveTypes.Uint64, `[1.8446744073709552e+19]`},
+ {"DurationFractional", arrow.FixedWidthTypes.Duration_s, `[1.5]`},
+ {"TimestampFractional", arrow.FixedWidthTypes.Timestamp_s, `[1.5]`},
+ {"Date32Fractional", arrow.FixedWidthTypes.Date32, `[1.5]`},
+ {"Date32OverflowPositive", arrow.FixedWidthTypes.Date32, `[2147483648]`},
+ {"Date32OverflowNegative", arrow.FixedWidthTypes.Date32, `[-2147483649]`},
+ {"Date64Fractional", arrow.FixedWidthTypes.Date64, `[1.5]`},
+ {"Time32Fractional", arrow.FixedWidthTypes.Time32s, `[1.5]`},
+ {"Time32OverflowPositive", arrow.FixedWidthTypes.Time32s, `[2147483648]`},
+ {"Time32OverflowNegative", arrow.FixedWidthTypes.Time32s, `[-2147483649]`},
+ {"Time64Fractional", arrow.FixedWidthTypes.Time64us, `[1.5]`},
+ {"Int64NaNString", arrow.PrimitiveTypes.Int64, `["NaN"]`},
+ {"Int64InfString", arrow.PrimitiveTypes.Int64, `["+Inf"]`},
+ {"Uint64NaNString", arrow.PrimitiveTypes.Uint64, `["NaN"]`},
+ {"Uint64InfString", arrow.PrimitiveTypes.Uint64, `["+Inf"]`},
+ {"Uint64ExponentialOverflowString", arrow.PrimitiveTypes.Uint64, `["1.8446744073709552e+19"]`},
+ {"Int64ExponentBeyondMantissa", arrow.PrimitiveTypes.Int64, `[9.007199254740993e15]`},
+ {"Uint64ExponentBeyondMantissa", arrow.PrimitiveTypes.Uint64, `[9.007199254740993e15]`},
+ {"Int64ExponentBeyondMantissaString", arrow.PrimitiveTypes.Int64, `["9.007199254740993e15"]`},
+ {"Uint64ExponentBeyondMantissaString", arrow.PrimitiveTypes.Uint64, `["9.007199254740993e15"]`},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
+ defer mem.AssertSize(t, 0)
+
+ arr, _, err := array.FromJSON(mem, tc.dt, strings.NewReader(tc.json))
+ if err == nil {
+ arr.Release()
+ t.Fatalf("expected error for %s with input %s, got nil", tc.name, tc.json)
+ }
+ })
+ }
+}
diff --git a/arrow/compute/arithmetic_test.go b/arrow/compute/arithmetic_test.go
index 07fb1fc9..67e22e8e 100644
--- a/arrow/compute/arithmetic_test.go
+++ b/arrow/compute/arithmetic_test.go
@@ -154,7 +154,7 @@ func (b *BinaryFuncTestSuite) TearDownTest() {
}
func (b *BinaryFuncTestSuite) getArr(dt arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(b.mem, dt, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(b.mem, dt, strings.NewReader(str))
b.Require().NoError(err)
return arr
}
@@ -164,9 +164,9 @@ type Float16BinaryFuncTestSuite struct {
}
func (b *Float16BinaryFuncTestSuite) assertBinopErr(fn binaryFunc, lhs, rhs string) {
- left, _, _ := array.FromJSON(b.mem, arrow.FixedWidthTypes.Float16, strings.NewReader(lhs), array.WithUseNumber())
+ left, _, _ := array.FromJSON(b.mem, arrow.FixedWidthTypes.Float16, strings.NewReader(lhs))
defer left.Release()
- right, _, _ := array.FromJSON(b.mem, arrow.FixedWidthTypes.Float16, strings.NewReader(rhs), array.WithUseNumber())
+ right, _, _ := array.FromJSON(b.mem, arrow.FixedWidthTypes.Float16, strings.NewReader(rhs))
defer right.Release()
_, err := fn(&compute.ArrayDatum{left.Data()}, &compute.ArrayDatum{right.Data()})
@@ -288,20 +288,20 @@ func (b *BinaryArithmeticSuite[T]) assertBinopArrs(fn binaryArithmeticFunc, lhs,
}
func (b *BinaryArithmeticSuite[T]) assertBinopExpArr(fn binaryArithmeticFunc, lhs, rhs string, exp arrow.Array) {
- left, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(lhs), array.WithUseNumber())
+ left, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(lhs))
defer left.Release()
- right, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(rhs), array.WithUseNumber())
+ right, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(rhs))
defer right.Release()
b.assertBinopArrs(fn, left, right, exp)
}
func (b *BinaryArithmeticSuite[T]) assertBinop(fn binaryArithmeticFunc, lhs, rhs, expected string) {
- left, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(lhs), array.WithUseNumber())
+ left, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(lhs))
defer left.Release()
- right, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(rhs), array.WithUseNumber())
+ right, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(rhs))
defer right.Release()
- exp, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(expected), array.WithUseNumber())
+ exp, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(expected))
defer exp.Release()
b.assertBinopArrs(fn, left, right, exp)
@@ -312,9 +312,9 @@ func (b *BinaryArithmeticSuite[T]) setOverflowCheck(value bool) {
}
func (b *BinaryArithmeticSuite[T]) assertBinopErr(fn binaryArithmeticFunc, lhs, rhs, expectedMsg string) {
- left, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(lhs), array.WithUseNumber())
+ left, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(lhs))
defer left.Release()
- right, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(rhs), array.WithUseNumber())
+ right, _, _ := array.FromJSON(b.mem, b.DataType(), strings.NewReader(rhs))
defer right.Release()
assertBinopErr(b.T(), func(left, right compute.Datum) (compute.Datum, error) {
@@ -2449,7 +2449,7 @@ func (us *UnaryArithmeticSuite[T, O]) makeArray(v ...T) arrow.Array {
}
func (us *UnaryArithmeticSuite[T, O]) getArr(dt arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(us.mem, dt, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(us.mem, dt, strings.NewReader(str))
us.Require().NoError(err)
return arr
}
@@ -2504,7 +2504,7 @@ func (us *UnaryArithmeticSuite[T, O]) assertUnaryOpArrs(fn unaryArithmeticFunc[O
}
func (us *UnaryArithmeticSuite[T, O]) assertUnaryOpExpArr(fn unaryArithmeticFunc[O], arg string, exp arrow.Array) {
- in, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(arg), array.WithUseNumber())
+ in, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(arg))
us.Require().NoError(err)
defer in.Release()
@@ -2512,10 +2512,10 @@ func (us *UnaryArithmeticSuite[T, O]) assertUnaryOpExpArr(fn unaryArithmeticFunc
}
func (us *UnaryArithmeticSuite[T, O]) assertUnaryOp(fn unaryArithmeticFunc[O], arg, exp string) {
- in, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(arg), array.WithUseNumber())
+ in, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(arg))
us.Require().NoError(err)
defer in.Release()
- expected, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(exp), array.WithUseNumber())
+ expected, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(exp))
us.Require().NoError(err)
defer expected.Release()
@@ -2523,7 +2523,7 @@ func (us *UnaryArithmeticSuite[T, O]) assertUnaryOp(fn unaryArithmeticFunc[O], a
}
func (us *UnaryArithmeticSuite[T, O]) assertUnaryOpErr(fn unaryArithmeticFunc[O], arg string, msg string) {
- in, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(arg), array.WithUseNumber())
+ in, _, err := array.FromJSON(us.mem, us.datatype(), strings.NewReader(arg))
us.Require().NoError(err)
defer in.Release()
diff --git a/arrow/compute/cast_test.go b/arrow/compute/cast_test.go
index a6d2e016..67420098 100644
--- a/arrow/compute/cast_test.go
+++ b/arrow/compute/cast_test.go
@@ -410,15 +410,15 @@ func (c *CastSuite) TestCanCast() {
}
func (c *CastSuite) checkCastFails(dt arrow.DataType, input string, opts *compute.CastOptions) {
- inArr, _, _ := array.FromJSON(c.mem, dt, strings.NewReader(input), array.WithUseNumber())
+ inArr, _, _ := array.FromJSON(c.mem, dt, strings.NewReader(input))
defer inArr.Release()
checkCastFails(c.T(), inArr, *opts)
}
func (c *CastSuite) checkCastOpts(dtIn, dtOut arrow.DataType, inJSON, outJSON string, opts compute.CastOptions) {
- inArr, _, _ := array.FromJSON(c.mem, dtIn, strings.NewReader(inJSON), array.WithUseNumber())
- outArr, _, _ := array.FromJSON(c.mem, dtOut, strings.NewReader(outJSON), array.WithUseNumber())
+ inArr, _, _ := array.FromJSON(c.mem, dtIn, strings.NewReader(inJSON))
+ outArr, _, _ := array.FromJSON(c.mem, dtOut, strings.NewReader(outJSON))
defer inArr.Release()
defer outArr.Release()
@@ -430,13 +430,13 @@ func (c *CastSuite) checkCast(dtIn, dtOut arrow.DataType, inJSON, outJSON string
}
func (c *CastSuite) checkCastArr(in arrow.Array, dtOut arrow.DataType, json string, opts compute.CastOptions) {
- outArr, _, _ := array.FromJSON(c.mem, dtOut, strings.NewReader(json), array.WithUseNumber())
+ outArr, _, _ := array.FromJSON(c.mem, dtOut, strings.NewReader(json))
defer outArr.Release()
checkCast(c.T(), in, outArr, opts)
}
func (c *CastSuite) checkCastExp(dtIn arrow.DataType, inJSON string, exp arrow.Array) {
- inArr, _, _ := array.FromJSON(c.mem, dtIn, strings.NewReader(inJSON), array.WithUseNumber())
+ inArr, _, _ := array.FromJSON(c.mem, dtIn, strings.NewReader(inJSON))
defer inArr.Release()
checkCast(c.T(), inArr, exp, *compute.DefaultCastOptions(true))
}
@@ -525,8 +525,7 @@ func (c *CastSuite) TestIntegerSignedToUnsigned() {
checkCast(c.T(), i32s, u32s, options)
u64s, _, _ := array.FromJSON(c.mem, arrow.PrimitiveTypes.Uint64,
- strings.NewReader(`[18446744071562067968, null, 18446744073709551615, 65535, 2147483647]`),
- array.WithUseNumber()) // have to use WithUseNumber so it doesn't lose precision converting to float64
+ strings.NewReader(`[18446744071562067968, null, 18446744073709551615, 65535, 2147483647]`))
defer u64s.Release()
checkCast(c.T(), i32s, u64s, options)
@@ -606,8 +605,7 @@ func (c *CastSuite) TestIntToFloating() {
}
i64s, _, _ := array.FromJSON(c.mem, arrow.PrimitiveTypes.Int64,
- strings.NewReader(`[-9223372036854775808, -9223372036854775807, 0, 9223372036854775806, 9223372036854775807]`),
- array.WithUseNumber())
+ strings.NewReader(`[-9223372036854775808, -9223372036854775807, 0, 9223372036854775806, 9223372036854775807]`))
defer i64s.Release()
checkCastFails(c.T(), i64s, *compute.SafeCastOptions(arrow.PrimitiveTypes.Float64))
@@ -666,7 +664,7 @@ func (c *CastSuite) TestDecimal128ToInt() {
strings.NewReader(`[
"12345678901234567890000.0000000000",
"99999999999999999999999.0000000000",
- null]`), array.WithUseNumber())
+ null]`))
defer overflowNoTrunc.Release()
opts.AllowIntOverflow = true
c.checkCastArr(overflowNoTrunc, arrow.PrimitiveTypes.Int64,
@@ -691,7 +689,7 @@ func (c *CastSuite) TestDecimal128ToInt() {
strings.NewReader(`[
"12345678901234567890000.0045345000",
"99999999999999999999999.0000344300",
- null]`), array.WithUseNumber())
+ null]`))
defer overflowAndTruncate.Release()
if opts.AllowIntOverflow && opts.AllowDecimalTruncate {
c.checkCastArr(overflowAndTruncate, arrow.PrimitiveTypes.Int64,
@@ -773,7 +771,7 @@ func (c *CastSuite) TestDecimal256ToInt() {
strings.NewReader(`[
"1234567890123456789000000.0000000000",
"9999999999999999999999999.0000000000",
- null]`), array.WithUseNumber())
+ null]`))
defer overflowNoTrunc.Release()
opts.AllowIntOverflow = true
c.checkCastArr(overflowNoTrunc, arrow.PrimitiveTypes.Int64,
@@ -798,7 +796,7 @@ func (c *CastSuite) TestDecimal256ToInt() {
strings.NewReader(`[
"1234567890123456789000000.0045345000",
"9999999999999999999999999.0000344300",
- null]`), array.WithUseNumber())
+ null]`))
defer overflowAndTruncate.Release()
if opts.AllowIntOverflow && opts.AllowDecimalTruncate {
c.checkCastArr(overflowAndTruncate, arrow.PrimitiveTypes.Int64,
@@ -1550,7 +1548,7 @@ func (c *CastSuite) TestBinaryLikeToBinaryView() {
})
defer exp.Release()
- in, _, err := array.FromJSON(c.mem, from, strings.NewReader(strInJSON), array.WithUseNumber())
+ in, _, err := array.FromJSON(c.mem, from, strings.NewReader(strInJSON))
c.Require().NoError(err)
defer in.Release()
@@ -1593,7 +1591,7 @@ func (c *CastSuite) TestBinaryLikeToBinaryView() {
c.Run("from fixed_size_binary", func() {
fsbType := &arrow.FixedSizeBinaryType{ByteWidth: 3}
in, _, err := array.FromJSON(c.mem, fsbType,
- strings.NewReader(`["YWJj", "ZGVm", null]`), array.WithUseNumber())
+ strings.NewReader(`["YWJj", "ZGVm", null]`))
c.Require().NoError(err)
defer in.Release()
@@ -2433,11 +2431,11 @@ func (c *CastSuite) TestBoolToStringViewStaysSingleBuffer() {
// scalar iteration path used by checkCast, which relies on scalar.GetScalar
// - that helper does not yet support view types (unrelated to GH-184).
func (c *CastSuite) checkCastArrayOnly(dtIn, dtOut arrow.DataType, inJSON, outJSON string) {
- inArr, _, err := array.FromJSON(c.mem, dtIn, strings.NewReader(inJSON), array.WithUseNumber())
+ inArr, _, err := array.FromJSON(c.mem, dtIn, strings.NewReader(inJSON))
c.Require().NoError(err)
defer inArr.Release()
- expArr, _, err := array.FromJSON(c.mem, dtOut, strings.NewReader(outJSON), array.WithUseNumber())
+ expArr, _, err := array.FromJSON(c.mem, dtOut, strings.NewReader(outJSON))
c.Require().NoError(err)
defer expArr.Release()
@@ -2672,7 +2670,7 @@ func (c *CastSuite) TestTimestampToDate() {
0, 951782400000, -2240524800000, 1999987200000,
1577836800000, 1577750400000, 1577664000000, 1262217600000,
1262304000000, 1262476800000, 1262563200000, 1136073600000,
- 1135987200000, 1230422400000, 1230508800000, 1325376000000, null]`), array.WithUseNumber())
+ 1135987200000, 1230422400000, 1230508800000, 1325376000000, null]`))
defer date64.Release()
checkCast(c.T(), stamps, date32, *compute.DefaultCastOptions(true))
diff --git a/arrow/compute/scalar_compare_test.go b/arrow/compute/scalar_compare_test.go
index 5a3078fc..a615145d 100644
--- a/arrow/compute/scalar_compare_test.go
+++ b/arrow/compute/scalar_compare_test.go
@@ -52,11 +52,11 @@ func (c *CompareSuite) validateCompareDatum(op kernels.CompareOperator, lhs, rhs
}
func (c *CompareSuite) validateCompare(op kernels.CompareOperator, dt arrow.DataType, lhsStr, rhsStr, expStr string) {
- lhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(lhsStr), array.WithUseNumber())
+ lhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(lhsStr))
c.Require().NoError(err)
- rhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(rhsStr), array.WithUseNumber())
+ rhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(rhsStr))
c.Require().NoError(err)
- exp, _, err := array.FromJSON(c.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(expStr), array.WithUseNumber())
+ exp, _, err := array.FromJSON(c.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(expStr))
c.Require().NoError(err)
defer func() {
lhs.Release()
@@ -67,9 +67,9 @@ func (c *CompareSuite) validateCompare(op kernels.CompareOperator, dt arrow.Data
}
func (c *CompareSuite) validateCompareArrScalar(op kernels.CompareOperator, dt arrow.DataType, lhsStr string, rhs compute.Datum, expStr string) {
- lhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(lhsStr), array.WithUseNumber())
+ lhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(lhsStr))
c.Require().NoError(err)
- exp, _, err := array.FromJSON(c.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(expStr), array.WithUseNumber())
+ exp, _, err := array.FromJSON(c.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(expStr))
c.Require().NoError(err)
defer func() {
lhs.Release()
@@ -79,9 +79,9 @@ func (c *CompareSuite) validateCompareArrScalar(op kernels.CompareOperator, dt a
}
func (c *CompareSuite) validateCompareScalarArr(op kernels.CompareOperator, dt arrow.DataType, lhs compute.Datum, rhsStr string, expStr string) {
- rhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(rhsStr), array.WithUseNumber())
+ rhs, _, err := array.FromJSON(c.mem, dt, strings.NewReader(rhsStr))
c.Require().NoError(err)
- exp, _, err := array.FromJSON(c.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(expStr), array.WithUseNumber())
+ exp, _, err := array.FromJSON(c.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(expStr))
c.Require().NoError(err)
defer func() {
rhs.Release()
@@ -1323,7 +1323,7 @@ func TestCompareGreaterWithImplicitCasts(t *testing.T) {
defer mem.AssertSize(t, 0)
getArr := func(ty arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(mem, ty, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(mem, ty, strings.NewReader(str))
require.NoError(t, err)
return arr
}
@@ -1377,7 +1377,7 @@ func TestCompareGreaterWithImplicitCastUint64EdgeCase(t *testing.T) {
defer mem.AssertSize(t, 0)
getArr := func(ty arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(mem, ty, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(mem, ty, strings.NewReader(str))
require.NoError(t, err)
return arr
}
@@ -1455,7 +1455,7 @@ func (sv *ScalarValiditySuite) TearDownTest() {
}
func (sv *ScalarValiditySuite) getArr(dt arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(sv.mem, dt, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(sv.mem, dt, strings.NewReader(str))
sv.Require().NoError(err)
return arr
}
diff --git a/arrow/compute/scalar_set_lookup_test.go b/arrow/compute/scalar_set_lookup_test.go
index 010bb52c..18b9a78e 100644
--- a/arrow/compute/scalar_set_lookup_test.go
+++ b/arrow/compute/scalar_set_lookup_test.go
@@ -44,7 +44,7 @@ func (ss *ScalarSetLookupSuite) SetupTest() {
}
func (ss *ScalarSetLookupSuite) getArr(dt arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(ss.mem, dt, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(ss.mem, dt, strings.NewReader(str))
ss.Require().NoError(err)
return arr
}
diff --git a/arrow/compute/vector_selection_test.go b/arrow/compute/vector_selection_test.go
index 62fcce08..5a25bfc3 100644
--- a/arrow/compute/vector_selection_test.go
+++ b/arrow/compute/vector_selection_test.go
@@ -61,7 +61,7 @@ func (f *FilterKernelTestSuite) TearDownTest() {
}
func (f *FilterKernelTestSuite) getArr(dt arrow.DataType, str string) arrow.Array {
- arr, _, err := array.FromJSON(f.mem, dt, strings.NewReader(str), array.WithUseNumber())
+ arr, _, err := array.FromJSON(f.mem, dt, strings.NewReader(str))
f.Require().NoError(err)
return arr
}
@@ -126,11 +126,11 @@ func (f *FilterKernelTestSuite) assertFilter(values, filter, expected arrow.Arra
}
func (f *FilterKernelTestSuite) assertFilterJSON(dt arrow.DataType, values, filter, expected string) {
- valuesArr, _, _ := array.FromJSON(f.mem, dt, strings.NewReader(values), array.WithUseNumber())
+ valuesArr, _, _ := array.FromJSON(f.mem, dt, strings.NewReader(values))
defer valuesArr.Release()
filterArr, _, _ := array.FromJSON(f.mem, arrow.FixedWidthTypes.Boolean, strings.NewReader(filter))
defer filterArr.Release()
- expectedArr, _, _ := array.FromJSON(f.mem, dt, strings.NewReader(expected), array.WithUseNumber())
+ expectedArr, _, _ := array.FromJSON(f.mem, dt, strings.NewReader(expected))
defer expectedArr.Release()
f.assertFilter(valuesArr, filterArr, expectedArr)
@@ -178,7 +178,7 @@ func (tk *TakeKernelTestSuite) assertTakeArrays(values, indices, expected arrow.
}
func (tk *TakeKernelTestSuite) takeJSON(dt arrow.DataType, values string, idxType arrow.DataType, indices string) (arrow.Array, error) {
- valArr, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(values), array.WithUseNumber())
+ valArr, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(values))
defer valArr.Release()
indArr, _, _ := array.FromJSON(tk.mem, idxType, strings.NewReader(indices))
defer indArr.Release()
@@ -187,9 +187,9 @@ func (tk *TakeKernelTestSuite) takeJSON(dt arrow.DataType, values string, idxTyp
}
func (tk *TakeKernelTestSuite) checkTake(dt arrow.DataType, valuesJSON, indicesJSON, expJSON string) {
- values, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(valuesJSON), array.WithUseNumber())
+ values, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(valuesJSON))
defer values.Release()
- expected, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(expJSON), array.WithUseNumber())
+ expected, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(expJSON))
defer expected.Release()
for _, idxType := range []arrow.DataType{arrow.PrimitiveTypes.Int8, arrow.PrimitiveTypes.Uint32} {
@@ -268,7 +268,7 @@ func (tk *TakeKernelTestSuite) assertNoValidityBitmapButUnknownNullCount(values,
}
func (tk *TakeKernelTestSuite) assertNoValidityBitmapUnknownNullCountJSON(dt arrow.DataType, values, indices string) {
- vals, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(values), array.WithUseNumber())
+ vals, _, _ := array.FromJSON(tk.mem, dt, strings.NewReader(values))
defer vals.Release()
inds, _, _ := array.FromJSON(tk.mem, arrow.PrimitiveTypes.Int16, strings.NewReader(indices))
defer inds.Release()
@@ -821,7 +821,7 @@ type FilterKernelWithRecordBatch struct {
}
func (f *FilterKernelWithRecordBatch) doFilter(sc *arrow.Schema, batchJSON, selection string, opts compute.FilterOptions) (arrow.RecordBatch, error) {
- rec, _, err := array.RecordFromJSON(f.mem, sc, strings.NewReader(batchJSON), array.WithUseNumber())
+ rec, _, err := array.RecordFromJSON(f.mem, sc, strings.NewReader(batchJSON))
if err != nil {
return nil, err
}
@@ -848,7 +848,7 @@ func (f *FilterKernelWithRecordBatch) assertFilter(sc *arrow.Schema, batchJSON,
f.Require().NoError(err)
defer actual.Release()
- expected, _, err := array.RecordFromJSON(f.mem, sc, strings.NewReader(expectedBatch), array.WithUseNumber())
+ expected, _, err := array.RecordFromJSON(f.mem, sc, strings.NewReader(expectedBatch))
f.Require().NoError(err)
defer expected.Release()
@@ -1781,7 +1781,7 @@ func TestMapPreservesValueNullable(t *testing.T) {
ctx := compute.WithAllocator(context.TODO(), mem)
dt := arrow.MapOf(arrow.BinaryTypes.String, arrow.PrimitiveTypes.Int32)
- values, _, _ := array.FromJSON(mem, dt, strings.NewReader(`[[{"key": "a", "value": 1}]]`), array.WithUseNumber())
+ values, _, _ := array.FromJSON(mem, dt, strings.NewReader(`[[{"key": "a", "value": 1}]]`))
defer values.Release()
indices, _, _ := array.FromJSON(mem, arrow.PrimitiveTypes.Int8, strings.NewReader(`[0]`))
defer indices.Release()
diff --git a/arrow/doc.go b/arrow/doc.go
index 6985543d..389d2fbd 100644
--- a/arrow/doc.go
+++ b/arrow/doc.go
@@ -36,7 +36,7 @@ package arrow
const PkgVersion = "18.6.0"
-//go:generate go run _tools/tmpl/main.go -i -data=numeric.tmpldata type_traits_numeric.gen.go.tmpl type_traits_numeric.gen_test.go.tmpl array/numeric.gen.go.tmpl array/numericbuilder.gen.go.tmpl array/bufferbuilder_numeric.gen.go.tmpl
+//go:generate go run _tools/tmpl/main.go -i -data=numeric.tmpldata type_traits_numeric.gen.go.tmpl type_traits_numeric.gen_test.go.tmpl array/numericbuilder.gen.go.tmpl array/numericbuilder.gen_test.go.tmpl array/bufferbuilder_numeric.gen.go.tmpl
//go:generate go run _tools/tmpl/main.go -i -data=datatype_numeric.gen.go.tmpldata datatype_numeric.gen.go.tmpl tensor/numeric.gen.go.tmpl tensor/numeric.gen_test.go.tmpl
//go:generate go run _tools/tmpl/main.go -i -data=scalar/numeric.gen.go.tmpldata scalar/numeric.gen.go.tmpl scalar/numeric.gen_test.go.tmpl
//go:generate go run ./gen-flatbuffers.go
diff --git a/arrow/flight/flightsql/example/type_info.go b/arrow/flight/flightsql/example/type_info.go
index d9a12f0f..60bbb069 100644
--- a/arrow/flight/flightsql/example/type_info.go
+++ b/arrow/flight/flightsql/example/type_info.go
@@ -62,7 +62,7 @@ func GetTypeInfoResult(mem memory.Allocator) arrow.RecordBatch {
// reference for creating a boolean() array with only zeros
zeroBoolArray, _, err := array.FromJSON(mem, arrow.FixedWidthTypes.Boolean,
- strings.NewReader(`[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]`), array.WithUseNumber())
+ strings.NewReader(`[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]`))
if err != nil {
panic(err)
}