You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Kent Wu <ke...@gmail.com> on 2026/07/02 20:37:27 UTC
[DISCUSS][Format] JSON representation of Arrow schemas
Hi all,
I'd like to raise a topic for discussion that has surfaced a few
separate times over the years in the Arrow community, which is the
lack of a canonical human-readable representation of Arrow schemas.
Arrow schemas today only canonically serialize as IPC binary, which is
a friction point for application-level tasks where binary blobs are
not ergonomic, such as JSON API contracts, hand authoring or reading,
and persistence use cases, to name a few.
The motivating pain point right now is at the ADBC metadata boundary.
The 1.2 milestone includes proposals for new metadata APIs, and
several contributors have noted that returning Arrow schemas via IPC
is unsatisfactory for this use case.
ADBC 1.2 Milestone
- https://github.com/apache/arrow-adbc/milestone/9
Discussion within various issues:
- https://github.com/apache/arrow-adbc/issues/4400
- https://github.com/apache/arrow-adbc/issues/1514
- https://github.com/apache/arrow-adbc/issues/1704
- https://github.com/apache/arrow-adbc/pull/4031
As a starting point, I've put together a proposal for how a JSON
representation might be structured.
Rather than dump the whole thing here, I've put it in this google doc
which is open for comments:
https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
It's early and open to change, so I'd welcome feedback of any kind.
Looking forward to hearing your thoughts
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Rusty Conover <ru...@query.farm>.
Reviewing the doc, specifically around dictionary encoding it was strange for me to see "dictionary" expressed at the same level of "type". Stylistically, I think it could be good to have the "type" property contain all of the type information including the dictionary encoding i.e. "type" set to "dictionary" just like it does for lists and structs.
Rusty
--
Rusty Conover
Query.Farm Founder
https://query.farm
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by David Li <li...@apache.org>.
Looks good to me. Thanks for the updates!
On Tue, Sep 29, 2026, at 13:31, Dewey Dunnington wrote:
> Hi Kent,
>
> Thank you for putting this revision together. I left a few minor comments
> in the document but in general this looks great and I am looking forward to
> implementing it and/or using it!
>
> Cheers,
>
> -dewey
>
> On Thu, Sep 17, 2026 at 10:06 PM Kent Wu <ke...@gmail.com> wrote:
>
>> Apologies for going quiet on this, and thank you Dewey, David, Rusty,
>> and Matt for the feedback.
>>
>> I incorporated some of the feedback into the doc [1], which I summarized
>> under draft-02.
>>
>> David:
>> > But I think "simple" string types is reasonable for unparameterized
>> > types.
>>
>> Agreed. I updated the doc with a proposed mapping of types represented
>> by simple string form vs object form. The mapping is unambiguous, so
>> each type has only one allowed representation (a type cannot have both a
>> valid simple string type and an object type).
>>
>> Matt:
>> > I wouldn't object to making metadata an object
>>
>> I agree, and incorporated it into the proposal. Surveying the ecosystem,
>> I found that many implementations expose metadata as a map, which cannot
>> represent duplicate keys. The object form aligns well with that.
>>
>> David:
>> > raises too many questions about what happens to the canonical
>> > representation/equality/serialization/round-tripping
>>
>> Most of these questions already exist and aren't specified by Arrow.
>> Equality and round-tripping for metadata aren't defined anywhere today,
>> and the escaped-JSON-in-JSON case is currently handled by
>> domain-specific convention at the application layer.
>>
>> After considering this and Dewey's feedback, I don't think this version
>> needs to define equality at all, either over documents or over schemas,
>> and I'm in agreement that the format should have an answer for
>> JSON-valued metadata.
>>
>> A specification of the mapping and a conformance suite are sufficient.
>> Losslessness is a property of a reader and writer pair, certified by the
>> suite. The suite has provenance the schemas themselves don't, since it
>> knows which metadata values a fixture wrote as a non-string, so it can
>> compare those as JSON and everything else byte-for-byte. Bindings don't
>> need to make that comparison - only the suite does, once, to certify the
>> pair round-trips a schema.
>>
>> To be clear about scope: this specifies the mapping between a document
>> and an Arrow schema, not the behavior of Arrow implementations. "reader"
>> and "writer" below mean the code performing that conversion in either
>> direction. What an implementation does with the metadata afterwards is
>> out of scope.
>>
>> This is the language I've proposed in the doc:
>>
>> A metadata value MAY be any JSON value. A reader MUST map a string
>> value to its contents. A reader MUST map a value of any other type
>> (object, array, number, boolean, or null) to its serialized string
>> form, for which the RFC 8785 serialization SHOULD be used. A writer
>> MAY emit a string or non-string value.
>>
>> Nothing here requires an implementation to detect which metadata values
>> are JSON, or to change its extension APIs. A writer that only emits
>> strings is fully conformant.
>>
>> I've left RFC 8785 [2] as a recommendation rather than a requirement.
>> The conformance requirement is that both forms map to the same Arrow
>> metadata, which the suite checks structurally.
>>
>> I'm not opposed to making it a requirement. I just don't have a good
>> sense of how much of a burden an RFC 8785 implementation is across
>> Arrow target languages.
>>
>> [1]:
>> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
>> [2]: https://www.rfc-editor.org/info/rfc8785/
>>
RE: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by "Lee, David via dev" <de...@arrow.apache.org>.
I'm using the c++ string aliases found here plus the ones I added to support decimal, binary, timestamp, lists, structs, union, etc.
https://github.com/apache/arrow/blob/main/python/pyarrow/types.pxi#L5764-L5820
Sample schema in json format:
{
"int_column": "int",
"int_column2": "int32",
"64bit_int_column": "int64",
"not_null_int_column": "int not null",
"nullable_int_column": "int null",
"string_column": "str",
"another_string_column": "string",
"dollar_value": "decimal(10,2)",
"phone_numbers": ["string"],
"address": {"city": "string", "street": "string", "house_no": "string"},
"pets": [{"name": "string", "animal_type": "string"}],
"union_column": ["string", "int"],
"list_of_union_column": [["string", "int"]]
}
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to https://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see https://www.blackrock.com/corporate/about-us/contacts-locations.
© 2026 BlackRock, Inc. All rights reserved.
RE: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by "Lee, David via dev" <de...@arrow.apache.org>.
I also created a JSON representation of an Arrow Schema here:
https://github.com/apache/arrow/issues/43858#issuecomment-2329537927
I'm using this format to define the data "contract" in HOCON/JSON config files to pass data in arrow format from one process to another..
columns = []
for field, typ in fields_dict.items():
pa_type = pa.gw_type_for_alias(typ) << enhanced version from the issue above to support nested data types, etc..
columns.append(pa.field(field, pa_type))
schema = pa.schema(columns)
return schema
-----Original Message-----
From: Lee, David
Sent: Tuesday, September 29, 2026 10:56 AM
To: dev@arrow.apache.org
Subject: RE: [DISCUSS][Format] JSON representation of Arrow schemas
I can't access the google docs behind a corporate firewall, but the JCS spec doesn't mention anything about Arrow Union types..
It would probably be best to include Arrow Sparse Union support for something like:
[
{"user_id": 123},
{"user_id": "Bob"}
]
I can't find the documentation, but I remember reading that arrow union types are written to parquet as structs, with metadata info so structs can be converted back into Arrow union types when reading parquet. We probably should adopt a similar metadata format so the above "user_id" object turns into an arrow sparse or dense union type. Dense unions have performance issues and probably should be avoided for a simpler format like json.
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to https://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see https://www.blackrock.com/corporate/about-us/contacts-locations.
© 2026 BlackRock, Inc. All rights reserved.
RE: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by "Lee, David via dev" <de...@arrow.apache.org>.
I can't access the google docs behind a corporate firewall, but the JCS spec doesn't mention anything about Arrow Union types..
It would probably be best to include Arrow Sparse Union support for something like:
[
{"user_id": 123},
{"user_id": "Bob"}
]
I can't find the documentation, but I remember reading that arrow union types are written to parquet as structs, with metadata info so structs can be converted back into Arrow union types when reading parquet. We probably should adopt a similar metadata format so the above "user_id" object turns into an arrow sparse or dense union type. Dense unions have performance issues and probably should be avoided for a simpler format like json.
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to https://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see https://www.blackrock.com/corporate/about-us/contacts-locations.
© 2026 BlackRock, Inc. All rights reserved.
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Dewey Dunnington <de...@gmail.com>.
Hi Kent,
Thank you for putting this revision together. I left a few minor comments
in the document but in general this looks great and I am looking forward to
implementing it and/or using it!
Cheers,
-dewey
On Thu, Sep 17, 2026 at 10:06 PM Kent Wu <ke...@gmail.com> wrote:
> Apologies for going quiet on this, and thank you Dewey, David, Rusty,
> and Matt for the feedback.
>
> I incorporated some of the feedback into the doc [1], which I summarized
> under draft-02.
>
> David:
> > But I think "simple" string types is reasonable for unparameterized
> > types.
>
> Agreed. I updated the doc with a proposed mapping of types represented
> by simple string form vs object form. The mapping is unambiguous, so
> each type has only one allowed representation (a type cannot have both a
> valid simple string type and an object type).
>
> Matt:
> > I wouldn't object to making metadata an object
>
> I agree, and incorporated it into the proposal. Surveying the ecosystem,
> I found that many implementations expose metadata as a map, which cannot
> represent duplicate keys. The object form aligns well with that.
>
> David:
> > raises too many questions about what happens to the canonical
> > representation/equality/serialization/round-tripping
>
> Most of these questions already exist and aren't specified by Arrow.
> Equality and round-tripping for metadata aren't defined anywhere today,
> and the escaped-JSON-in-JSON case is currently handled by
> domain-specific convention at the application layer.
>
> After considering this and Dewey's feedback, I don't think this version
> needs to define equality at all, either over documents or over schemas,
> and I'm in agreement that the format should have an answer for
> JSON-valued metadata.
>
> A specification of the mapping and a conformance suite are sufficient.
> Losslessness is a property of a reader and writer pair, certified by the
> suite. The suite has provenance the schemas themselves don't, since it
> knows which metadata values a fixture wrote as a non-string, so it can
> compare those as JSON and everything else byte-for-byte. Bindings don't
> need to make that comparison - only the suite does, once, to certify the
> pair round-trips a schema.
>
> To be clear about scope: this specifies the mapping between a document
> and an Arrow schema, not the behavior of Arrow implementations. "reader"
> and "writer" below mean the code performing that conversion in either
> direction. What an implementation does with the metadata afterwards is
> out of scope.
>
> This is the language I've proposed in the doc:
>
> A metadata value MAY be any JSON value. A reader MUST map a string
> value to its contents. A reader MUST map a value of any other type
> (object, array, number, boolean, or null) to its serialized string
> form, for which the RFC 8785 serialization SHOULD be used. A writer
> MAY emit a string or non-string value.
>
> Nothing here requires an implementation to detect which metadata values
> are JSON, or to change its extension APIs. A writer that only emits
> strings is fully conformant.
>
> I've left RFC 8785 [2] as a recommendation rather than a requirement.
> The conformance requirement is that both forms map to the same Arrow
> metadata, which the suite checks structurally.
>
> I'm not opposed to making it a requirement. I just don't have a good
> sense of how much of a burden an RFC 8785 implementation is across
> Arrow target languages.
>
> [1]:
> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
> [2]: https://www.rfc-editor.org/info/rfc8785/
>
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Kent Wu <ke...@gmail.com>.
Apologies for going quiet on this, and thank you Dewey, David, Rusty,
and Matt for the feedback.
I incorporated some of the feedback into the doc [1], which I summarized
under draft-02.
David:
> But I think "simple" string types is reasonable for unparameterized
> types.
Agreed. I updated the doc with a proposed mapping of types represented
by simple string form vs object form. The mapping is unambiguous, so
each type has only one allowed representation (a type cannot have both a
valid simple string type and an object type).
Matt:
> I wouldn't object to making metadata an object
I agree, and incorporated it into the proposal. Surveying the ecosystem,
I found that many implementations expose metadata as a map, which cannot
represent duplicate keys. The object form aligns well with that.
David:
> raises too many questions about what happens to the canonical
> representation/equality/serialization/round-tripping
Most of these questions already exist and aren't specified by Arrow.
Equality and round-tripping for metadata aren't defined anywhere today,
and the escaped-JSON-in-JSON case is currently handled by
domain-specific convention at the application layer.
After considering this and Dewey's feedback, I don't think this version
needs to define equality at all, either over documents or over schemas,
and I'm in agreement that the format should have an answer for
JSON-valued metadata.
A specification of the mapping and a conformance suite are sufficient.
Losslessness is a property of a reader and writer pair, certified by the
suite. The suite has provenance the schemas themselves don't, since it
knows which metadata values a fixture wrote as a non-string, so it can
compare those as JSON and everything else byte-for-byte. Bindings don't
need to make that comparison - only the suite does, once, to certify the
pair round-trips a schema.
To be clear about scope: this specifies the mapping between a document
and an Arrow schema, not the behavior of Arrow implementations. "reader"
and "writer" below mean the code performing that conversion in either
direction. What an implementation does with the metadata afterwards is
out of scope.
This is the language I've proposed in the doc:
A metadata value MAY be any JSON value. A reader MUST map a string
value to its contents. A reader MUST map a value of any other type
(object, array, number, boolean, or null) to its serialized string
form, for which the RFC 8785 serialization SHOULD be used. A writer
MAY emit a string or non-string value.
Nothing here requires an implementation to detect which metadata values
are JSON, or to change its extension APIs. A writer that only emits
strings is fully conformant.
I've left RFC 8785 [2] as a recommendation rather than a requirement.
The conformance requirement is that both forms map to the same Arrow
metadata, which the suite checks structurally.
I'm not opposed to making it a requirement. I just don't have a good
sense of how much of a burden an RFC 8785 implementation is across
Arrow target languages.
[1]: https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
[2]: https://www.rfc-editor.org/info/rfc8785/
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Dewey Dunnington <de...@gmail.com>.
> we either have to complicate how we define equality or the JSON
representation can't round
> trip without significant special casing for implementations with extension
> types which we're trying to avoid
The equality of JSON objects within a metadata value JSON string is never
guaranteed, nor is roundtripping (e.g., Arrow C++ would instantiate an
ExtensionType instance and reserialize it, potentially putting keys in a
different order). Enabling metadata values to be any valid JSON would
improve both equality and roundtripping (because the JSON object, not its
stringified version, could be compared for equality). The extra complexity
is on readers, who when they see a metadata value that is not a string have
to serialize it as one (this is what we do for GeoArrow -> Parquet and I
believe it's a two-liner). Writers aren't required to carry extra
complexity, they just aren't prevented from writing idiomatic JSON if they
would like to (they could be warned that such values may be serialized on
roundtrip if that is important to a specific producer). Many writers
already special case extension types on both import and export to
generate/parse field metadata...this is not a new code path.
> I wouldn't object to making metadata an object.
If the goal is idiomatic JSON, I think this would be a good choice.
Cheers,
-dewey
On Sat, Jul 11, 2026 at 12:41 PM Matt Topol <zo...@gmail.com> wrote:
> > The language here could be similar...implementations are not required to
> serialize them any differently but are allowed to produce nicer looking
> JSON for extension types when they can guarantee it's valid (many
> implementations can do this, at least for canonical extensions whose
> serialization has been voted on and is part of the spec, sort of).
>
> The problem i have with this is just that it means we either have to
> complicate how we define equality or the JSON representation can't round
> trip without significant special casing for implementations with extension
> types which we're trying to avoid. I just don't think the convenience of
> nicer looking JSON is worth the complexity. i.e. I agree with what David
> said earlier in this thread.
>
> > All that said, just making "metadata" an object instead of a list of key
> values leads to pretty equivalent extraction of an extension parameter
>
> We discussed this a bit and landed on using the list of key values to be
> more like the IPC representation, but I wouldn't object to making metadata
> an object.
>
> --Matt
>
> On Fri, Jul 10, 2026, 10:47 PM Dewey Dunnington <
> dewey.dunnington@gmail.com>
> wrote:
>
> > > To practically implement inline JSON for extension types, though
> >
> > The way we do this in GeoArrow (Arrow Go, Arrow C++ Parquet,
> > arrow-rs/parquet) to prevent escaped-json-in-json when going from Parquet
> > (crs is a string) -> GeoArrow (crs is a member of a JSON object) is to
> > parse the value, and if it parses as a JSON object, use that instead of
> the
> > stringified version. The language we have in the spec for this is
> > "Producers should not write an escaped JSON object to the crs key" [1].
> The
> > language here could be similar...implementations are not required to
> > serialize them any differently but are allowed to produce nicer looking
> > JSON for extension types when they can guarantee it's valid (many
> > implementations can do this, at least for canonical extensions whose
> > serialization has been voted on and is part of the spec, sort of).
> >
> > > but only if implementations are required to do this
> >
> > While I would actually love lifting extensions in implementations having
> > spent quite a lot of time tracking down over- and under- propagated field
> > metadata, I don't think where the extension name and metadata live in the
> > JSON structure matters for this: deserializers that have a registry can
> > query it a bit more easily, and deserializers that don't can put it in
> > field metadata. Serializers would have to special-case those two metadata
> > keys. All that said, just making "metadata" an object instead of a list
> of
> > key values leads to pretty equivalent extraction of an extension
> parameter
> > (f["metadata"]["ARROW:extension:metadata"]["crs"] vs
> > f["type"]["metadata"]["crs"], or something).
> >
> > > But I think "simple" string types is reasonable for unparameterized
> > types.
> >
> > I hope we do go this direction...I dug up the testing JSON reader for the
> > last time I had to parse that [2] and while it's not terrible, it's also
> > not great if we can avoid it.
> >
> > [1] https://geoarrow.org/extension-types.html#extension-metadata
> > [2]
> >
> >
> https://github.com/apache/arrow-nanoarrow/blob/a5d7011ff4838bd4cbf70c6f72b7078ea8842cd4/src/nanoarrow/testing/testing.cc#L1044-L1099
> >
> > On Fri, Jul 10, 2026 at 5:54 PM David Li <li...@apache.org> wrote:
> >
> > > To practically implement inline JSON for extension types, though, would
> > we
> > > be changing (breaking?) all the extension APIs in Arrow implementations
> > to
> > > somehow flag that they use JSON? And in any case, if some Arrow
> > > implementation doesn't handle this, or the extension isn't registered,
> > etc.
> > > you could still end up having a string instead of JSON and would need
> to
> > > handle it right? I don't think implementations will guarantee that JSON
> > > metadata always gets serialized as JSON.
> > >
> > > Hence while it seems convenient I'd like to avoid trying to
> special-case
> > > extension types and JSON.
> > >
> > > I could be convinced that extension types should be lifted to the type
> > and
> > > not the metadata, but only if implementations are required to do this
> (so
> > > no extension type metadata in the metadata, ever) again because the
> > > inconsistency would otherwise make things annoying (I already run into
> > this
> > > with Arrow implementations where only registered extension types get
> > > promoted to actual types, leading to duplicate logic/checking all over
> > the
> > > place).
> > >
> > > (FWIW: Java does actually enumerate bitwidth and signedness explicitly;
> > it
> > > also puts child fields on the field and not the type, both choices
> > matching
> > > Flatbuffers exactly. But I think "simple" string types is reasonable
> for
> > > unparameterized types.)
> > >
> > > On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote:
> > > > For at least the "uint32" abbreviation, this is how most if not all
> > > > implementations actually enumerate the type (not integer +
> signedness +
> > > > bitwidth), so I think that particular change makes the parsers
> simpler
> > in
> > > > addition to being more compact.
> > > >
> > > > For extension types that use JSON, this requires a very awkward path
> to
> > > > inspecting the type, perhaps before an Arrow implementation is
> > involved.
> > > > Maybe
> > > >
> > > > "type": {"name": "extension", "extension:name": "geoarrow.wkb",
> > > > "extension:metadata": {...}}
> > > >
> > > > ...would make API responses that contain extension types sufficiently
> > > > compact that a flat list of named key/value objects doesn't matter.
> > With
> > > > the metadata as an object instead of a list and the value field as
> any
> > > JSON
> > > > value I think it's still sufficiently compact to pluck an extension
> > > > parameter. The alternative is sufficiently gross (loop through named
> > key
> > > > and pick the first? or last? and re-parse escaped JSON?) that I am
> not
> > > sure
> > > > I would actually use it on the day I need to return a schema with
> > > GeoArrow
> > > > types in a JSON API response (also: this day may never come :))
> > > >
> > > > -dewey
> > > >
> > > > On Wed, Jul 8, 2026 at 9:16 PM David Li <li...@apache.org> wrote:
> > > >
> > > >> I think there's value, though, in having a representation that is
> (1)
> > > >> unambiguous and consistent (2) reasonably human-friendly and
> > > >> machine-friendly, even if it's not the most compact. This would be
> > > useful
> > > >> for (REST) APIs and ADBC, and I don't think the compactness of the
> > > >> representation is as important there. What kinds of use cases are
> you
> > > >> envisioning?
> > > >>
> > > >> FWIW, we debated the JSON metadata question - I kind of wanted this
> > too,
> > > >> but it raises too many questions about what happens to the canonical
> > > >> representation/equality/serialization/round-tripping. (For example:
> we
> > > need
> > > >> to stringify the JSON to pass it to the type parser, and a
> round-trip
> > > would
> > > >> possibly present semantically equivalent JSON but different JSON
> > > strings.
> > > >> JSON doesn't _disallow_ duplicate keys, so what if we get different
> > > >> behavior? etc.)
> > > >>
> > > >> Ultimately the motivation (for me) is to use this in ADBC where we
> > want
> > > to
> > > >> represent Arrow schemas inside Arrow data (where serialized IPC
> > schemas
> > > >> seem to be unpalatable), and so being well-defined is most important
> > (I
> > > >> don't really want to handroll something that is only used in ADBC).
> > > >>
> > > >> -David
> > > >>
> > > >> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
> > > >> > Thank you for drafting!
> > > >> >
> > > >> > I gave a few comments in the community call last Wednesday, but
> I'll
> > > >> > echo them here for discussion. None of these are blocking
> concerns,
> > > >> > just thoughts having seen this particular concept be reinvented
> in a
> > > >> > lot of places.
> > > >> >
> > > >> > While this does a great job of losslessly translating flatbuffer
> > > >> > schemas into JSON and back again, flatbuffer schemas are rather
> > > >> > awkwardly laid out and the human typability/readability is not
> > great.
> > > >> > I think there is an opportunity for this JSON spec to replace a
> lot
> > of
> > > >> > bespoke data type / field / schema representations, but I worry
> that
> > > >> > it's a little verbose in its current form to fill that niche.
> > > >> >
> > > >> > Just as an example, a schema with a single uint32 column with this
> > > >> proposal is:
> > > >> >
> > > >> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable":
> > true,
> > > >> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
> > > >> >
> > > >> > ...whereas with a bit of language (apply defaults for nullability,
> > > >> > optional/independent versioning, special case unparameterized
> types)
> > > >> > this could be more like:
> > > >> >
> > > >> > [{"name": "col0", "type": "uint32"}]
> > > >> >
> > > >> > ...drawing from BigQuery [1] here), although that does open up a
> > > >> > significantly more surface area for bikeshedding on which strings
> > map
> > > >> > to which type (unless we want to reuse the C data interface ones,
> > > >> > which from the proposal it seems like maybe we don't).
> > > >> >
> > > >> > Another opportunity to reduce verbosity is metadata, which for an
> > > >> > extension type with JSON metadata would be
> > > >> >
> > > >> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
> > > >> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
> > > >> >
> > > >> > ...and could be more readable:
> > > >> >
> > > >> > {"ARROW:extension:name": "geoarrow.wkb",
> "ARROW:extension:metadata":
> > > >> > {"crs": "OGC:CRS84"}}
> > > >> >
> > > >> > Basically, if the goal is human readability, I think it's worth
> the
> > > >> > effort to do the bikeshedding on how to represent these concepts
> > that
> > > >> > mirror how they are printed/enumerated in implementations (not
> > > >> > necessarily the .fbs files). If the goal is machine readability
> > within
> > > >> > existing arrow implementations, perfectly mirroring flatbuffers
> JSON
> > > >> > or the C data interface would be less work (more opportunity for
> > > >> > reusing existing functions).
> > > >> >
> > > >> > Cheers,
> > > >> >
> > > >> > -dewey
> > > >> >
> > > >> > [1]
> > > >> >
> > > >>
> > >
> >
> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
> > > >> >
> > > >> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com>
> wrote:
> > > >> >>
> > > >> >> Hi all,
> > > >> >>
> > > >> >> I'd like to raise a topic for discussion that has surfaced a few
> > > >> >> separate times over the years in the Arrow community, which is
> the
> > > >> >> lack of a canonical human-readable representation of Arrow
> schemas.
> > > >> >>
> > > >> >> Arrow schemas today only canonically serialize as IPC binary,
> which
> > > is
> > > >> >> a friction point for application-level tasks where binary blobs
> are
> > > >> >> not ergonomic, such as JSON API contracts, hand authoring or
> > reading,
> > > >> >> and persistence use cases, to name a few.
> > > >> >>
> > > >> >> The motivating pain point right now is at the ADBC metadata
> > boundary.
> > > >> >> The 1.2 milestone includes proposals for new metadata APIs, and
> > > >> >> several contributors have noted that returning Arrow schemas via
> > IPC
> > > >> >> is unsatisfactory for this use case.
> > > >> >>
> > > >> >> ADBC 1.2 Milestone
> > > >> >> - https://github.com/apache/arrow-adbc/milestone/9
> > > >> >>
> > > >> >> Discussion within various issues:
> > > >> >> - https://github.com/apache/arrow-adbc/issues/4400
> > > >> >> - https://github.com/apache/arrow-adbc/issues/1514
> > > >> >> - https://github.com/apache/arrow-adbc/issues/1704
> > > >> >> - https://github.com/apache/arrow-adbc/pull/4031
> > > >> >>
> > > >> >> As a starting point, I've put together a proposal for how a JSON
> > > >> >> representation might be structured.
> > > >> >>
> > > >> >> Rather than dump the whole thing here, I've put it in this google
> > doc
> > > >> >> which is open for comments:
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
> > > >> >>
> > > >> >> It's early and open to change, so I'd welcome feedback of any
> kind.
> > > >> >>
> > > >> >> Looking forward to hearing your thoughts
> > > >>
> > >
> >
>
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Matt Topol <zo...@gmail.com>.
> The language here could be similar...implementations are not required to
serialize them any differently but are allowed to produce nicer looking
JSON for extension types when they can guarantee it's valid (many
implementations can do this, at least for canonical extensions whose
serialization has been voted on and is part of the spec, sort of).
The problem i have with this is just that it means we either have to
complicate how we define equality or the JSON representation can't round
trip without significant special casing for implementations with extension
types which we're trying to avoid. I just don't think the convenience of
nicer looking JSON is worth the complexity. i.e. I agree with what David
said earlier in this thread.
> All that said, just making "metadata" an object instead of a list of key
values leads to pretty equivalent extraction of an extension parameter
We discussed this a bit and landed on using the list of key values to be
more like the IPC representation, but I wouldn't object to making metadata
an object.
--Matt
On Fri, Jul 10, 2026, 10:47 PM Dewey Dunnington <de...@gmail.com>
wrote:
> > To practically implement inline JSON for extension types, though
>
> The way we do this in GeoArrow (Arrow Go, Arrow C++ Parquet,
> arrow-rs/parquet) to prevent escaped-json-in-json when going from Parquet
> (crs is a string) -> GeoArrow (crs is a member of a JSON object) is to
> parse the value, and if it parses as a JSON object, use that instead of the
> stringified version. The language we have in the spec for this is
> "Producers should not write an escaped JSON object to the crs key" [1]. The
> language here could be similar...implementations are not required to
> serialize them any differently but are allowed to produce nicer looking
> JSON for extension types when they can guarantee it's valid (many
> implementations can do this, at least for canonical extensions whose
> serialization has been voted on and is part of the spec, sort of).
>
> > but only if implementations are required to do this
>
> While I would actually love lifting extensions in implementations having
> spent quite a lot of time tracking down over- and under- propagated field
> metadata, I don't think where the extension name and metadata live in the
> JSON structure matters for this: deserializers that have a registry can
> query it a bit more easily, and deserializers that don't can put it in
> field metadata. Serializers would have to special-case those two metadata
> keys. All that said, just making "metadata" an object instead of a list of
> key values leads to pretty equivalent extraction of an extension parameter
> (f["metadata"]["ARROW:extension:metadata"]["crs"] vs
> f["type"]["metadata"]["crs"], or something).
>
> > But I think "simple" string types is reasonable for unparameterized
> types.
>
> I hope we do go this direction...I dug up the testing JSON reader for the
> last time I had to parse that [2] and while it's not terrible, it's also
> not great if we can avoid it.
>
> [1] https://geoarrow.org/extension-types.html#extension-metadata
> [2]
>
> https://github.com/apache/arrow-nanoarrow/blob/a5d7011ff4838bd4cbf70c6f72b7078ea8842cd4/src/nanoarrow/testing/testing.cc#L1044-L1099
>
> On Fri, Jul 10, 2026 at 5:54 PM David Li <li...@apache.org> wrote:
>
> > To practically implement inline JSON for extension types, though, would
> we
> > be changing (breaking?) all the extension APIs in Arrow implementations
> to
> > somehow flag that they use JSON? And in any case, if some Arrow
> > implementation doesn't handle this, or the extension isn't registered,
> etc.
> > you could still end up having a string instead of JSON and would need to
> > handle it right? I don't think implementations will guarantee that JSON
> > metadata always gets serialized as JSON.
> >
> > Hence while it seems convenient I'd like to avoid trying to special-case
> > extension types and JSON.
> >
> > I could be convinced that extension types should be lifted to the type
> and
> > not the metadata, but only if implementations are required to do this (so
> > no extension type metadata in the metadata, ever) again because the
> > inconsistency would otherwise make things annoying (I already run into
> this
> > with Arrow implementations where only registered extension types get
> > promoted to actual types, leading to duplicate logic/checking all over
> the
> > place).
> >
> > (FWIW: Java does actually enumerate bitwidth and signedness explicitly;
> it
> > also puts child fields on the field and not the type, both choices
> matching
> > Flatbuffers exactly. But I think "simple" string types is reasonable for
> > unparameterized types.)
> >
> > On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote:
> > > For at least the "uint32" abbreviation, this is how most if not all
> > > implementations actually enumerate the type (not integer + signedness +
> > > bitwidth), so I think that particular change makes the parsers simpler
> in
> > > addition to being more compact.
> > >
> > > For extension types that use JSON, this requires a very awkward path to
> > > inspecting the type, perhaps before an Arrow implementation is
> involved.
> > > Maybe
> > >
> > > "type": {"name": "extension", "extension:name": "geoarrow.wkb",
> > > "extension:metadata": {...}}
> > >
> > > ...would make API responses that contain extension types sufficiently
> > > compact that a flat list of named key/value objects doesn't matter.
> With
> > > the metadata as an object instead of a list and the value field as any
> > JSON
> > > value I think it's still sufficiently compact to pluck an extension
> > > parameter. The alternative is sufficiently gross (loop through named
> key
> > > and pick the first? or last? and re-parse escaped JSON?) that I am not
> > sure
> > > I would actually use it on the day I need to return a schema with
> > GeoArrow
> > > types in a JSON API response (also: this day may never come :))
> > >
> > > -dewey
> > >
> > > On Wed, Jul 8, 2026 at 9:16 PM David Li <li...@apache.org> wrote:
> > >
> > >> I think there's value, though, in having a representation that is (1)
> > >> unambiguous and consistent (2) reasonably human-friendly and
> > >> machine-friendly, even if it's not the most compact. This would be
> > useful
> > >> for (REST) APIs and ADBC, and I don't think the compactness of the
> > >> representation is as important there. What kinds of use cases are you
> > >> envisioning?
> > >>
> > >> FWIW, we debated the JSON metadata question - I kind of wanted this
> too,
> > >> but it raises too many questions about what happens to the canonical
> > >> representation/equality/serialization/round-tripping. (For example: we
> > need
> > >> to stringify the JSON to pass it to the type parser, and a round-trip
> > would
> > >> possibly present semantically equivalent JSON but different JSON
> > strings.
> > >> JSON doesn't _disallow_ duplicate keys, so what if we get different
> > >> behavior? etc.)
> > >>
> > >> Ultimately the motivation (for me) is to use this in ADBC where we
> want
> > to
> > >> represent Arrow schemas inside Arrow data (where serialized IPC
> schemas
> > >> seem to be unpalatable), and so being well-defined is most important
> (I
> > >> don't really want to handroll something that is only used in ADBC).
> > >>
> > >> -David
> > >>
> > >> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
> > >> > Thank you for drafting!
> > >> >
> > >> > I gave a few comments in the community call last Wednesday, but I'll
> > >> > echo them here for discussion. None of these are blocking concerns,
> > >> > just thoughts having seen this particular concept be reinvented in a
> > >> > lot of places.
> > >> >
> > >> > While this does a great job of losslessly translating flatbuffer
> > >> > schemas into JSON and back again, flatbuffer schemas are rather
> > >> > awkwardly laid out and the human typability/readability is not
> great.
> > >> > I think there is an opportunity for this JSON spec to replace a lot
> of
> > >> > bespoke data type / field / schema representations, but I worry that
> > >> > it's a little verbose in its current form to fill that niche.
> > >> >
> > >> > Just as an example, a schema with a single uint32 column with this
> > >> proposal is:
> > >> >
> > >> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable":
> true,
> > >> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
> > >> >
> > >> > ...whereas with a bit of language (apply defaults for nullability,
> > >> > optional/independent versioning, special case unparameterized types)
> > >> > this could be more like:
> > >> >
> > >> > [{"name": "col0", "type": "uint32"}]
> > >> >
> > >> > ...drawing from BigQuery [1] here), although that does open up a
> > >> > significantly more surface area for bikeshedding on which strings
> map
> > >> > to which type (unless we want to reuse the C data interface ones,
> > >> > which from the proposal it seems like maybe we don't).
> > >> >
> > >> > Another opportunity to reduce verbosity is metadata, which for an
> > >> > extension type with JSON metadata would be
> > >> >
> > >> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
> > >> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
> > >> >
> > >> > ...and could be more readable:
> > >> >
> > >> > {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
> > >> > {"crs": "OGC:CRS84"}}
> > >> >
> > >> > Basically, if the goal is human readability, I think it's worth the
> > >> > effort to do the bikeshedding on how to represent these concepts
> that
> > >> > mirror how they are printed/enumerated in implementations (not
> > >> > necessarily the .fbs files). If the goal is machine readability
> within
> > >> > existing arrow implementations, perfectly mirroring flatbuffers JSON
> > >> > or the C data interface would be less work (more opportunity for
> > >> > reusing existing functions).
> > >> >
> > >> > Cheers,
> > >> >
> > >> > -dewey
> > >> >
> > >> > [1]
> > >> >
> > >>
> >
> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
> > >> >
> > >> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com> wrote:
> > >> >>
> > >> >> Hi all,
> > >> >>
> > >> >> I'd like to raise a topic for discussion that has surfaced a few
> > >> >> separate times over the years in the Arrow community, which is the
> > >> >> lack of a canonical human-readable representation of Arrow schemas.
> > >> >>
> > >> >> Arrow schemas today only canonically serialize as IPC binary, which
> > is
> > >> >> a friction point for application-level tasks where binary blobs are
> > >> >> not ergonomic, such as JSON API contracts, hand authoring or
> reading,
> > >> >> and persistence use cases, to name a few.
> > >> >>
> > >> >> The motivating pain point right now is at the ADBC metadata
> boundary.
> > >> >> The 1.2 milestone includes proposals for new metadata APIs, and
> > >> >> several contributors have noted that returning Arrow schemas via
> IPC
> > >> >> is unsatisfactory for this use case.
> > >> >>
> > >> >> ADBC 1.2 Milestone
> > >> >> - https://github.com/apache/arrow-adbc/milestone/9
> > >> >>
> > >> >> Discussion within various issues:
> > >> >> - https://github.com/apache/arrow-adbc/issues/4400
> > >> >> - https://github.com/apache/arrow-adbc/issues/1514
> > >> >> - https://github.com/apache/arrow-adbc/issues/1704
> > >> >> - https://github.com/apache/arrow-adbc/pull/4031
> > >> >>
> > >> >> As a starting point, I've put together a proposal for how a JSON
> > >> >> representation might be structured.
> > >> >>
> > >> >> Rather than dump the whole thing here, I've put it in this google
> doc
> > >> >> which is open for comments:
> > >> >>
> > >> >>
> > >>
> >
> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
> > >> >>
> > >> >> It's early and open to change, so I'd welcome feedback of any kind.
> > >> >>
> > >> >> Looking forward to hearing your thoughts
> > >>
> >
>
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Dewey Dunnington <de...@gmail.com>.
> To practically implement inline JSON for extension types, though
The way we do this in GeoArrow (Arrow Go, Arrow C++ Parquet,
arrow-rs/parquet) to prevent escaped-json-in-json when going from Parquet
(crs is a string) -> GeoArrow (crs is a member of a JSON object) is to
parse the value, and if it parses as a JSON object, use that instead of the
stringified version. The language we have in the spec for this is
"Producers should not write an escaped JSON object to the crs key" [1]. The
language here could be similar...implementations are not required to
serialize them any differently but are allowed to produce nicer looking
JSON for extension types when they can guarantee it's valid (many
implementations can do this, at least for canonical extensions whose
serialization has been voted on and is part of the spec, sort of).
> but only if implementations are required to do this
While I would actually love lifting extensions in implementations having
spent quite a lot of time tracking down over- and under- propagated field
metadata, I don't think where the extension name and metadata live in the
JSON structure matters for this: deserializers that have a registry can
query it a bit more easily, and deserializers that don't can put it in
field metadata. Serializers would have to special-case those two metadata
keys. All that said, just making "metadata" an object instead of a list of
key values leads to pretty equivalent extraction of an extension parameter
(f["metadata"]["ARROW:extension:metadata"]["crs"] vs
f["type"]["metadata"]["crs"], or something).
> But I think "simple" string types is reasonable for unparameterized types.
I hope we do go this direction...I dug up the testing JSON reader for the
last time I had to parse that [2] and while it's not terrible, it's also
not great if we can avoid it.
[1] https://geoarrow.org/extension-types.html#extension-metadata
[2]
https://github.com/apache/arrow-nanoarrow/blob/a5d7011ff4838bd4cbf70c6f72b7078ea8842cd4/src/nanoarrow/testing/testing.cc#L1044-L1099
On Fri, Jul 10, 2026 at 5:54 PM David Li <li...@apache.org> wrote:
> To practically implement inline JSON for extension types, though, would we
> be changing (breaking?) all the extension APIs in Arrow implementations to
> somehow flag that they use JSON? And in any case, if some Arrow
> implementation doesn't handle this, or the extension isn't registered, etc.
> you could still end up having a string instead of JSON and would need to
> handle it right? I don't think implementations will guarantee that JSON
> metadata always gets serialized as JSON.
>
> Hence while it seems convenient I'd like to avoid trying to special-case
> extension types and JSON.
>
> I could be convinced that extension types should be lifted to the type and
> not the metadata, but only if implementations are required to do this (so
> no extension type metadata in the metadata, ever) again because the
> inconsistency would otherwise make things annoying (I already run into this
> with Arrow implementations where only registered extension types get
> promoted to actual types, leading to duplicate logic/checking all over the
> place).
>
> (FWIW: Java does actually enumerate bitwidth and signedness explicitly; it
> also puts child fields on the field and not the type, both choices matching
> Flatbuffers exactly. But I think "simple" string types is reasonable for
> unparameterized types.)
>
> On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote:
> > For at least the "uint32" abbreviation, this is how most if not all
> > implementations actually enumerate the type (not integer + signedness +
> > bitwidth), so I think that particular change makes the parsers simpler in
> > addition to being more compact.
> >
> > For extension types that use JSON, this requires a very awkward path to
> > inspecting the type, perhaps before an Arrow implementation is involved.
> > Maybe
> >
> > "type": {"name": "extension", "extension:name": "geoarrow.wkb",
> > "extension:metadata": {...}}
> >
> > ...would make API responses that contain extension types sufficiently
> > compact that a flat list of named key/value objects doesn't matter. With
> > the metadata as an object instead of a list and the value field as any
> JSON
> > value I think it's still sufficiently compact to pluck an extension
> > parameter. The alternative is sufficiently gross (loop through named key
> > and pick the first? or last? and re-parse escaped JSON?) that I am not
> sure
> > I would actually use it on the day I need to return a schema with
> GeoArrow
> > types in a JSON API response (also: this day may never come :))
> >
> > -dewey
> >
> > On Wed, Jul 8, 2026 at 9:16 PM David Li <li...@apache.org> wrote:
> >
> >> I think there's value, though, in having a representation that is (1)
> >> unambiguous and consistent (2) reasonably human-friendly and
> >> machine-friendly, even if it's not the most compact. This would be
> useful
> >> for (REST) APIs and ADBC, and I don't think the compactness of the
> >> representation is as important there. What kinds of use cases are you
> >> envisioning?
> >>
> >> FWIW, we debated the JSON metadata question - I kind of wanted this too,
> >> but it raises too many questions about what happens to the canonical
> >> representation/equality/serialization/round-tripping. (For example: we
> need
> >> to stringify the JSON to pass it to the type parser, and a round-trip
> would
> >> possibly present semantically equivalent JSON but different JSON
> strings.
> >> JSON doesn't _disallow_ duplicate keys, so what if we get different
> >> behavior? etc.)
> >>
> >> Ultimately the motivation (for me) is to use this in ADBC where we want
> to
> >> represent Arrow schemas inside Arrow data (where serialized IPC schemas
> >> seem to be unpalatable), and so being well-defined is most important (I
> >> don't really want to handroll something that is only used in ADBC).
> >>
> >> -David
> >>
> >> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
> >> > Thank you for drafting!
> >> >
> >> > I gave a few comments in the community call last Wednesday, but I'll
> >> > echo them here for discussion. None of these are blocking concerns,
> >> > just thoughts having seen this particular concept be reinvented in a
> >> > lot of places.
> >> >
> >> > While this does a great job of losslessly translating flatbuffer
> >> > schemas into JSON and back again, flatbuffer schemas are rather
> >> > awkwardly laid out and the human typability/readability is not great.
> >> > I think there is an opportunity for this JSON spec to replace a lot of
> >> > bespoke data type / field / schema representations, but I worry that
> >> > it's a little verbose in its current form to fill that niche.
> >> >
> >> > Just as an example, a schema with a single uint32 column with this
> >> proposal is:
> >> >
> >> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": true,
> >> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
> >> >
> >> > ...whereas with a bit of language (apply defaults for nullability,
> >> > optional/independent versioning, special case unparameterized types)
> >> > this could be more like:
> >> >
> >> > [{"name": "col0", "type": "uint32"}]
> >> >
> >> > ...drawing from BigQuery [1] here), although that does open up a
> >> > significantly more surface area for bikeshedding on which strings map
> >> > to which type (unless we want to reuse the C data interface ones,
> >> > which from the proposal it seems like maybe we don't).
> >> >
> >> > Another opportunity to reduce verbosity is metadata, which for an
> >> > extension type with JSON metadata would be
> >> >
> >> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
> >> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
> >> >
> >> > ...and could be more readable:
> >> >
> >> > {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
> >> > {"crs": "OGC:CRS84"}}
> >> >
> >> > Basically, if the goal is human readability, I think it's worth the
> >> > effort to do the bikeshedding on how to represent these concepts that
> >> > mirror how they are printed/enumerated in implementations (not
> >> > necessarily the .fbs files). If the goal is machine readability within
> >> > existing arrow implementations, perfectly mirroring flatbuffers JSON
> >> > or the C data interface would be less work (more opportunity for
> >> > reusing existing functions).
> >> >
> >> > Cheers,
> >> >
> >> > -dewey
> >> >
> >> > [1]
> >> >
> >>
> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
> >> >
> >> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com> wrote:
> >> >>
> >> >> Hi all,
> >> >>
> >> >> I'd like to raise a topic for discussion that has surfaced a few
> >> >> separate times over the years in the Arrow community, which is the
> >> >> lack of a canonical human-readable representation of Arrow schemas.
> >> >>
> >> >> Arrow schemas today only canonically serialize as IPC binary, which
> is
> >> >> a friction point for application-level tasks where binary blobs are
> >> >> not ergonomic, such as JSON API contracts, hand authoring or reading,
> >> >> and persistence use cases, to name a few.
> >> >>
> >> >> The motivating pain point right now is at the ADBC metadata boundary.
> >> >> The 1.2 milestone includes proposals for new metadata APIs, and
> >> >> several contributors have noted that returning Arrow schemas via IPC
> >> >> is unsatisfactory for this use case.
> >> >>
> >> >> ADBC 1.2 Milestone
> >> >> - https://github.com/apache/arrow-adbc/milestone/9
> >> >>
> >> >> Discussion within various issues:
> >> >> - https://github.com/apache/arrow-adbc/issues/4400
> >> >> - https://github.com/apache/arrow-adbc/issues/1514
> >> >> - https://github.com/apache/arrow-adbc/issues/1704
> >> >> - https://github.com/apache/arrow-adbc/pull/4031
> >> >>
> >> >> As a starting point, I've put together a proposal for how a JSON
> >> >> representation might be structured.
> >> >>
> >> >> Rather than dump the whole thing here, I've put it in this google doc
> >> >> which is open for comments:
> >> >>
> >> >>
> >>
> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
> >> >>
> >> >> It's early and open to change, so I'd welcome feedback of any kind.
> >> >>
> >> >> Looking forward to hearing your thoughts
> >>
>
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by David Li <li...@apache.org>.
To practically implement inline JSON for extension types, though, would we be changing (breaking?) all the extension APIs in Arrow implementations to somehow flag that they use JSON? And in any case, if some Arrow implementation doesn't handle this, or the extension isn't registered, etc. you could still end up having a string instead of JSON and would need to handle it right? I don't think implementations will guarantee that JSON metadata always gets serialized as JSON.
Hence while it seems convenient I'd like to avoid trying to special-case extension types and JSON.
I could be convinced that extension types should be lifted to the type and not the metadata, but only if implementations are required to do this (so no extension type metadata in the metadata, ever) again because the inconsistency would otherwise make things annoying (I already run into this with Arrow implementations where only registered extension types get promoted to actual types, leading to duplicate logic/checking all over the place).
(FWIW: Java does actually enumerate bitwidth and signedness explicitly; it also puts child fields on the field and not the type, both choices matching Flatbuffers exactly. But I think "simple" string types is reasonable for unparameterized types.)
On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote:
> For at least the "uint32" abbreviation, this is how most if not all
> implementations actually enumerate the type (not integer + signedness +
> bitwidth), so I think that particular change makes the parsers simpler in
> addition to being more compact.
>
> For extension types that use JSON, this requires a very awkward path to
> inspecting the type, perhaps before an Arrow implementation is involved.
> Maybe
>
> "type": {"name": "extension", "extension:name": "geoarrow.wkb",
> "extension:metadata": {...}}
>
> ...would make API responses that contain extension types sufficiently
> compact that a flat list of named key/value objects doesn't matter. With
> the metadata as an object instead of a list and the value field as any JSON
> value I think it's still sufficiently compact to pluck an extension
> parameter. The alternative is sufficiently gross (loop through named key
> and pick the first? or last? and re-parse escaped JSON?) that I am not sure
> I would actually use it on the day I need to return a schema with GeoArrow
> types in a JSON API response (also: this day may never come :))
>
> -dewey
>
> On Wed, Jul 8, 2026 at 9:16 PM David Li <li...@apache.org> wrote:
>
>> I think there's value, though, in having a representation that is (1)
>> unambiguous and consistent (2) reasonably human-friendly and
>> machine-friendly, even if it's not the most compact. This would be useful
>> for (REST) APIs and ADBC, and I don't think the compactness of the
>> representation is as important there. What kinds of use cases are you
>> envisioning?
>>
>> FWIW, we debated the JSON metadata question - I kind of wanted this too,
>> but it raises too many questions about what happens to the canonical
>> representation/equality/serialization/round-tripping. (For example: we need
>> to stringify the JSON to pass it to the type parser, and a round-trip would
>> possibly present semantically equivalent JSON but different JSON strings.
>> JSON doesn't _disallow_ duplicate keys, so what if we get different
>> behavior? etc.)
>>
>> Ultimately the motivation (for me) is to use this in ADBC where we want to
>> represent Arrow schemas inside Arrow data (where serialized IPC schemas
>> seem to be unpalatable), and so being well-defined is most important (I
>> don't really want to handroll something that is only used in ADBC).
>>
>> -David
>>
>> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
>> > Thank you for drafting!
>> >
>> > I gave a few comments in the community call last Wednesday, but I'll
>> > echo them here for discussion. None of these are blocking concerns,
>> > just thoughts having seen this particular concept be reinvented in a
>> > lot of places.
>> >
>> > While this does a great job of losslessly translating flatbuffer
>> > schemas into JSON and back again, flatbuffer schemas are rather
>> > awkwardly laid out and the human typability/readability is not great.
>> > I think there is an opportunity for this JSON spec to replace a lot of
>> > bespoke data type / field / schema representations, but I worry that
>> > it's a little verbose in its current form to fill that niche.
>> >
>> > Just as an example, a schema with a single uint32 column with this
>> proposal is:
>> >
>> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": true,
>> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
>> >
>> > ...whereas with a bit of language (apply defaults for nullability,
>> > optional/independent versioning, special case unparameterized types)
>> > this could be more like:
>> >
>> > [{"name": "col0", "type": "uint32"}]
>> >
>> > ...drawing from BigQuery [1] here), although that does open up a
>> > significantly more surface area for bikeshedding on which strings map
>> > to which type (unless we want to reuse the C data interface ones,
>> > which from the proposal it seems like maybe we don't).
>> >
>> > Another opportunity to reduce verbosity is metadata, which for an
>> > extension type with JSON metadata would be
>> >
>> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
>> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
>> >
>> > ...and could be more readable:
>> >
>> > {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
>> > {"crs": "OGC:CRS84"}}
>> >
>> > Basically, if the goal is human readability, I think it's worth the
>> > effort to do the bikeshedding on how to represent these concepts that
>> > mirror how they are printed/enumerated in implementations (not
>> > necessarily the .fbs files). If the goal is machine readability within
>> > existing arrow implementations, perfectly mirroring flatbuffers JSON
>> > or the C data interface would be less work (more opportunity for
>> > reusing existing functions).
>> >
>> > Cheers,
>> >
>> > -dewey
>> >
>> > [1]
>> >
>> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
>> >
>> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I'd like to raise a topic for discussion that has surfaced a few
>> >> separate times over the years in the Arrow community, which is the
>> >> lack of a canonical human-readable representation of Arrow schemas.
>> >>
>> >> Arrow schemas today only canonically serialize as IPC binary, which is
>> >> a friction point for application-level tasks where binary blobs are
>> >> not ergonomic, such as JSON API contracts, hand authoring or reading,
>> >> and persistence use cases, to name a few.
>> >>
>> >> The motivating pain point right now is at the ADBC metadata boundary.
>> >> The 1.2 milestone includes proposals for new metadata APIs, and
>> >> several contributors have noted that returning Arrow schemas via IPC
>> >> is unsatisfactory for this use case.
>> >>
>> >> ADBC 1.2 Milestone
>> >> - https://github.com/apache/arrow-adbc/milestone/9
>> >>
>> >> Discussion within various issues:
>> >> - https://github.com/apache/arrow-adbc/issues/4400
>> >> - https://github.com/apache/arrow-adbc/issues/1514
>> >> - https://github.com/apache/arrow-adbc/issues/1704
>> >> - https://github.com/apache/arrow-adbc/pull/4031
>> >>
>> >> As a starting point, I've put together a proposal for how a JSON
>> >> representation might be structured.
>> >>
>> >> Rather than dump the whole thing here, I've put it in this google doc
>> >> which is open for comments:
>> >>
>> >>
>> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
>> >>
>> >> It's early and open to change, so I'd welcome feedback of any kind.
>> >>
>> >> Looking forward to hearing your thoughts
>>
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Dewey Dunnington <de...@gmail.com>.
For at least the "uint32" abbreviation, this is how most if not all
implementations actually enumerate the type (not integer + signedness +
bitwidth), so I think that particular change makes the parsers simpler in
addition to being more compact.
For extension types that use JSON, this requires a very awkward path to
inspecting the type, perhaps before an Arrow implementation is involved.
Maybe
"type": {"name": "extension", "extension:name": "geoarrow.wkb",
"extension:metadata": {...}}
...would make API responses that contain extension types sufficiently
compact that a flat list of named key/value objects doesn't matter. With
the metadata as an object instead of a list and the value field as any JSON
value I think it's still sufficiently compact to pluck an extension
parameter. The alternative is sufficiently gross (loop through named key
and pick the first? or last? and re-parse escaped JSON?) that I am not sure
I would actually use it on the day I need to return a schema with GeoArrow
types in a JSON API response (also: this day may never come :))
-dewey
On Wed, Jul 8, 2026 at 9:16 PM David Li <li...@apache.org> wrote:
> I think there's value, though, in having a representation that is (1)
> unambiguous and consistent (2) reasonably human-friendly and
> machine-friendly, even if it's not the most compact. This would be useful
> for (REST) APIs and ADBC, and I don't think the compactness of the
> representation is as important there. What kinds of use cases are you
> envisioning?
>
> FWIW, we debated the JSON metadata question - I kind of wanted this too,
> but it raises too many questions about what happens to the canonical
> representation/equality/serialization/round-tripping. (For example: we need
> to stringify the JSON to pass it to the type parser, and a round-trip would
> possibly present semantically equivalent JSON but different JSON strings.
> JSON doesn't _disallow_ duplicate keys, so what if we get different
> behavior? etc.)
>
> Ultimately the motivation (for me) is to use this in ADBC where we want to
> represent Arrow schemas inside Arrow data (where serialized IPC schemas
> seem to be unpalatable), and so being well-defined is most important (I
> don't really want to handroll something that is only used in ADBC).
>
> -David
>
> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
> > Thank you for drafting!
> >
> > I gave a few comments in the community call last Wednesday, but I'll
> > echo them here for discussion. None of these are blocking concerns,
> > just thoughts having seen this particular concept be reinvented in a
> > lot of places.
> >
> > While this does a great job of losslessly translating flatbuffer
> > schemas into JSON and back again, flatbuffer schemas are rather
> > awkwardly laid out and the human typability/readability is not great.
> > I think there is an opportunity for this JSON spec to replace a lot of
> > bespoke data type / field / schema representations, but I worry that
> > it's a little verbose in its current form to fill that niche.
> >
> > Just as an example, a schema with a single uint32 column with this
> proposal is:
> >
> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": true,
> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
> >
> > ...whereas with a bit of language (apply defaults for nullability,
> > optional/independent versioning, special case unparameterized types)
> > this could be more like:
> >
> > [{"name": "col0", "type": "uint32"}]
> >
> > ...drawing from BigQuery [1] here), although that does open up a
> > significantly more surface area for bikeshedding on which strings map
> > to which type (unless we want to reuse the C data interface ones,
> > which from the proposal it seems like maybe we don't).
> >
> > Another opportunity to reduce verbosity is metadata, which for an
> > extension type with JSON metadata would be
> >
> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
> >
> > ...and could be more readable:
> >
> > {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
> > {"crs": "OGC:CRS84"}}
> >
> > Basically, if the goal is human readability, I think it's worth the
> > effort to do the bikeshedding on how to represent these concepts that
> > mirror how they are printed/enumerated in implementations (not
> > necessarily the .fbs files). If the goal is machine readability within
> > existing arrow implementations, perfectly mirroring flatbuffers JSON
> > or the C data interface would be less work (more opportunity for
> > reusing existing functions).
> >
> > Cheers,
> >
> > -dewey
> >
> > [1]
> >
> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
> >
> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> I'd like to raise a topic for discussion that has surfaced a few
> >> separate times over the years in the Arrow community, which is the
> >> lack of a canonical human-readable representation of Arrow schemas.
> >>
> >> Arrow schemas today only canonically serialize as IPC binary, which is
> >> a friction point for application-level tasks where binary blobs are
> >> not ergonomic, such as JSON API contracts, hand authoring or reading,
> >> and persistence use cases, to name a few.
> >>
> >> The motivating pain point right now is at the ADBC metadata boundary.
> >> The 1.2 milestone includes proposals for new metadata APIs, and
> >> several contributors have noted that returning Arrow schemas via IPC
> >> is unsatisfactory for this use case.
> >>
> >> ADBC 1.2 Milestone
> >> - https://github.com/apache/arrow-adbc/milestone/9
> >>
> >> Discussion within various issues:
> >> - https://github.com/apache/arrow-adbc/issues/4400
> >> - https://github.com/apache/arrow-adbc/issues/1514
> >> - https://github.com/apache/arrow-adbc/issues/1704
> >> - https://github.com/apache/arrow-adbc/pull/4031
> >>
> >> As a starting point, I've put together a proposal for how a JSON
> >> representation might be structured.
> >>
> >> Rather than dump the whole thing here, I've put it in this google doc
> >> which is open for comments:
> >>
> >>
> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
> >>
> >> It's early and open to change, so I'd welcome feedback of any kind.
> >>
> >> Looking forward to hearing your thoughts
>
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by David Li <li...@apache.org>.
I think there's value, though, in having a representation that is (1) unambiguous and consistent (2) reasonably human-friendly and machine-friendly, even if it's not the most compact. This would be useful for (REST) APIs and ADBC, and I don't think the compactness of the representation is as important there. What kinds of use cases are you envisioning?
FWIW, we debated the JSON metadata question - I kind of wanted this too, but it raises too many questions about what happens to the canonical representation/equality/serialization/round-tripping. (For example: we need to stringify the JSON to pass it to the type parser, and a round-trip would possibly present semantically equivalent JSON but different JSON strings. JSON doesn't _disallow_ duplicate keys, so what if we get different behavior? etc.)
Ultimately the motivation (for me) is to use this in ADBC where we want to represent Arrow schemas inside Arrow data (where serialized IPC schemas seem to be unpalatable), and so being well-defined is most important (I don't really want to handroll something that is only used in ADBC).
-David
On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote:
> Thank you for drafting!
>
> I gave a few comments in the community call last Wednesday, but I'll
> echo them here for discussion. None of these are blocking concerns,
> just thoughts having seen this particular concept be reinvented in a
> lot of places.
>
> While this does a great job of losslessly translating flatbuffer
> schemas into JSON and back again, flatbuffer schemas are rather
> awkwardly laid out and the human typability/readability is not great.
> I think there is an opportunity for this JSON spec to replace a lot of
> bespoke data type / field / schema representations, but I worry that
> it's a little verbose in its current form to fill that niche.
>
> Just as an example, a schema with a single uint32 column with this proposal is:
>
> {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": true,
> "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
>
> ...whereas with a bit of language (apply defaults for nullability,
> optional/independent versioning, special case unparameterized types)
> this could be more like:
>
> [{"name": "col0", "type": "uint32"}]
>
> ...drawing from BigQuery [1] here), although that does open up a
> significantly more surface area for bikeshedding on which strings map
> to which type (unless we want to reuse the C data interface ones,
> which from the proposal it seems like maybe we don't).
>
> Another opportunity to reduce verbosity is metadata, which for an
> extension type with JSON metadata would be
>
> [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
> "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
>
> ...and could be more readable:
>
> {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
> {"crs": "OGC:CRS84"}}
>
> Basically, if the goal is human readability, I think it's worth the
> effort to do the bikeshedding on how to represent these concepts that
> mirror how they are printed/enumerated in implementations (not
> necessarily the .fbs files). If the goal is machine readability within
> existing arrow implementations, perfectly mirroring flatbuffers JSON
> or the C data interface would be less work (more opportunity for
> reusing existing functions).
>
> Cheers,
>
> -dewey
>
> [1]
> https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
>
> On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I'd like to raise a topic for discussion that has surfaced a few
>> separate times over the years in the Arrow community, which is the
>> lack of a canonical human-readable representation of Arrow schemas.
>>
>> Arrow schemas today only canonically serialize as IPC binary, which is
>> a friction point for application-level tasks where binary blobs are
>> not ergonomic, such as JSON API contracts, hand authoring or reading,
>> and persistence use cases, to name a few.
>>
>> The motivating pain point right now is at the ADBC metadata boundary.
>> The 1.2 milestone includes proposals for new metadata APIs, and
>> several contributors have noted that returning Arrow schemas via IPC
>> is unsatisfactory for this use case.
>>
>> ADBC 1.2 Milestone
>> - https://github.com/apache/arrow-adbc/milestone/9
>>
>> Discussion within various issues:
>> - https://github.com/apache/arrow-adbc/issues/4400
>> - https://github.com/apache/arrow-adbc/issues/1514
>> - https://github.com/apache/arrow-adbc/issues/1704
>> - https://github.com/apache/arrow-adbc/pull/4031
>>
>> As a starting point, I've put together a proposal for how a JSON
>> representation might be structured.
>>
>> Rather than dump the whole thing here, I've put it in this google doc
>> which is open for comments:
>>
>> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
>>
>> It's early and open to change, so I'd welcome feedback of any kind.
>>
>> Looking forward to hearing your thoughts
Re: [DISCUSS][Format] JSON representation of Arrow schemas
Posted by Dewey Dunnington <de...@gmail.com>.
Thank you for drafting!
I gave a few comments in the community call last Wednesday, but I'll
echo them here for discussion. None of these are blocking concerns,
just thoughts having seen this particular concept be reinvented in a
lot of places.
While this does a great job of losslessly translating flatbuffer
schemas into JSON and back again, flatbuffer schemas are rather
awkwardly laid out and the human typability/readability is not great.
I think there is an opportunity for this JSON spec to replace a lot of
bespoke data type / field / schema representations, but I worry that
it's a little verbose in its current form to fill that niche.
Just as an example, a schema with a single uint32 column with this proposal is:
{"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": true,
"type": {"name": "int", "bitWidth": 32, "isSigned": false}}]}
...whereas with a bit of language (apply defaults for nullability,
optional/independent versioning, special case unparameterized types)
this could be more like:
[{"name": "col0", "type": "uint32"}]
...drawing from BigQuery [1] here), although that does open up a
significantly more surface area for bikeshedding on which strings map
to which type (unless we want to reuse the C data interface ones,
which from the proposal it seems like maybe we don't).
Another opportunity to reduce verbosity is metadata, which for an
extension type with JSON metadata would be
[{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key":
"ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}]
...and could be more readable:
{"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata":
{"crs": "OGC:CRS84"}}
Basically, if the goal is human readability, I think it's worth the
effort to do the bikeshedding on how to represent these concepts that
mirror how they are printed/enumerated in implementations (not
necessarily the .fbs files). If the goal is machine readability within
existing arrow implementations, perfectly mirroring flatbuffers JSON
or the C data interface would be less work (more opportunity for
reusing existing functions).
Cheers,
-dewey
[1] https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_
On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <ke...@gmail.com> wrote:
>
> Hi all,
>
> I'd like to raise a topic for discussion that has surfaced a few
> separate times over the years in the Arrow community, which is the
> lack of a canonical human-readable representation of Arrow schemas.
>
> Arrow schemas today only canonically serialize as IPC binary, which is
> a friction point for application-level tasks where binary blobs are
> not ergonomic, such as JSON API contracts, hand authoring or reading,
> and persistence use cases, to name a few.
>
> The motivating pain point right now is at the ADBC metadata boundary.
> The 1.2 milestone includes proposals for new metadata APIs, and
> several contributors have noted that returning Arrow schemas via IPC
> is unsatisfactory for this use case.
>
> ADBC 1.2 Milestone
> - https://github.com/apache/arrow-adbc/milestone/9
>
> Discussion within various issues:
> - https://github.com/apache/arrow-adbc/issues/4400
> - https://github.com/apache/arrow-adbc/issues/1514
> - https://github.com/apache/arrow-adbc/issues/1704
> - https://github.com/apache/arrow-adbc/pull/4031
>
> As a starting point, I've put together a proposal for how a JSON
> representation might be structured.
>
> Rather than dump the whole thing here, I've put it in this google doc
> which is open for comments:
>
> https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing
>
> It's early and open to change, so I'd welcome feedback of any kind.
>
> Looking forward to hearing your thoughts