You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2024/04/17 13:29:51 UTC

[PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

rok opened a new pull request, #41257:
URL: https://github.com/apache/arrow/pull/41257

   ### Rationale for this change
   
   As per #41256 this proposes a specification of a canonical extension type for a generic text data format that will enable storing text based formats such as JSON, JAML and XML.
   
   ### What changes are included in this PR?
   
   This adds to documentation of canonical extension types.
   
   ### Are these changes tested?
   
   No as only docs are changed.
   
   ### Are there any user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061337916

   By the way, the notion of a "generic" extension type meant to represent arbitrary third-party data is similar to the ["other / unsupported data type"](https://lists.apache.org/thread/8d5ldl5cb7mms21rd15lhpfrv4j9no4n) currently being discussed on the ML. @lidavidm @westonpace 
   
   I don't see any reason to restrict such an "other" type to text data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061402822

   Sur, but where did you get that idea of a "generic type"?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585081438


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   I think I agree with Ian that stricter is better.  It's fine for different components within a system to agree they want to support a wider set of values.  However, once data leaves a solution, it should assume that most components will only be processing the to-spec JSON.  This document is primarily about interchange and so should be concerned with the latter case (data leaving a solution to be consumed by some unknown component).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571018013


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   I don't see a downside allowing for `LargeStringArray` even though it doesn't make much sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592665683


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -51,7 +51,7 @@ types:
 
   3) Its serialization *must* be described in the proposal and should
      not require unduly implementation work or unusual software dependencies
-     (for example, a trivial custom text format or JSON would be acceptable).
+     (for example, a trivial custom text format or YAML would be acceptable).

Review Comment:
   This sentence is **not** an example of a canonical extension type. It describes how canonical extension types _may_ be serialized as metadata.
   
   So this should remain JSON. We would definitely not allow YAML as the serialization format for canonical extension types.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571011302


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,24 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is an empty string.

Review Comment:
   Year, that makes sense as we are promising [backwards compatibility](https://arrow.apache.org/docs/format/CanonicalExtensions.html#making-modifications):
   
   > Making Modifications
   > Like standard Arrow data types, canonical extension types should be considered stable once standardized. Modifying a canonical extension type (for example to expand the set of parameters) should be an exceptional event, follow the same rules as laid out above, and provide backwards compatibility guarantees.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585499203


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   @progger-dev is 1. or 2. like what you have in mind?
   1 . `Non-conformant or non-parsable JSON values should be considered invalid and halt producing or consuming process.`
   2. `Non-conformant or non-parsable JSON values should be considered as null values by producers and parsers.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1583441040


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   I think the one thing to consider here as possible future extensions (this might have already been discussed on ML a while ago), is non-standard encodings using Binary types instead of String.
   
   In the short term we should probably recommend that implementations should likely reject Binary annotated JSON fields



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1569069770


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is an empty string.
+  - Example: ``""``

Review Comment:
   It's great. It covers all cases. Removing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571026818


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,26 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.

Review Comment:
   @pitrou 
   ```suggestion
   * The storage type of this extension is ``StringArray`` or
     ``StringViewArray``. Only UTF-8 encoded JSON is supported.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571508109


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,26 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.

Review Comment:
   Should one of those say "LargeString"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1570995188


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,24 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is an empty string.

Review Comment:
   That would be interesting, though I'm not sure about adding that until we have someone who will definitely implement using it.
   
   We could phrase this in a way that metadata is currently optional. Maybe:
   
   ```suggestion
     Metadata is either an empty string or a JSON string with an empty object.
     In the future, additional fields may be added, but they are not required
     to interpret the array.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573281316


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   Is there a specific reason we want to add this here for JSON, while we don't do this for uuid? Do we expect there to be metadata in the future?
   
   (I would also argue that adding _optional_ metadata in the future should be regarded as backwards compatible, even if we don't specify that here now) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585935670


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   > I think we should also add language that specifies what happens when you receive non-conformant JSON or see any kind of parse errors. I think it's important for components that sit in between producers and consumers (and thus act as both).
   
   Is it needed to specify anything about that? To me that seems a decision that an application that will actually use the JSON data will need or want to decide for themselves how to treat invalid input, while in many cases an Arrow implementation receiving such data might just be passing it along, and will typically not do any validation at all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1570967434


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   > the Large variants can perhaps be omitted
   
   Some Arrow-native libraries (such as Polars, before it switched to use `StringView`) might work exclusively work large variants. I'm inclined to include them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061520224

   It was proposed on the [ML](https://lists.apache.org/thread/p3353oz6lk846pnoq6vk638tjqz2hm1j).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573298088


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   @tustvold suggested metadata could hold JSON schema which would not be the case with UUID and could justify the difference between the types.
   I do agree with your reasoning that adding optional metadata later wouldn't be backwards incompatible, but perhaps we have users that need it now. @progger-dev any thoughts from you or someone on the bigquery team?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585219769


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   Added rfc8259 requirement (https://github.com/apache/arrow/pull/41257/commits/594945010e3b7d393b411aad971743ffcdbdbc8e).
   
   To the original @jorisvandenbossche's question: it seems to me we're more in favour of adding optional metadata to JSON type than not so I'm leaving that language as is. Please correct me if I'm wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "progger-dev (via GitHub)" <gi...@apache.org>.
progger-dev commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585672675


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   I was thinking about treating it the same as if you didn't support the extension type itself and just dealt with the underlying type and acts as a pass through.
   
   But on second thought, this is not something for the specification as its likely going to be highly implementation dependent. Maybe its more of a recommendation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585269512


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON as specified in `rfc8259`_ is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.

Review Comment:
   In addition to being an empty string, is it also allowed to be just absent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592671363


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: ``arrow.json``.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.

Review Comment:
   One further nit: "StringArray" is not really generic terminology that we use in the format specs (it's the name of the array/vector in Arrow C++ and pyarrow, but not necessarily in other implementations). So I think you can leave out the `Array` part in each of those three types. 
   
   (strictly speaking the spec actually uses Utf8 / LargeUtf8 / Utf8View, but I think in many places we do use "String" instead)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592666250


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -51,7 +51,7 @@ types:
 
   3) Its serialization *must* be described in the proposal and should
      not require unduly implementation work or unusual software dependencies
-     (for example, a trivial custom text format or JSON would be acceptable).
+     (for example, a trivial custom text format or YAML would be acceptable).

Review Comment:
   @ianmcook 's suggestion of "a JSON-based format" is correct though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061561335

   @pitrou this is now fully JSONified. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585434862


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON as specified in `rfc8259`_ is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.

Review Comment:
   For now I think we should think of the extension serialization metadata key  as mandatory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2064252327

   This makes sense to me. I did have a potentially stupid thought that perhaps we just define a canonical "MIME" extension type, and that way avoid having to define lots of MIME types, but then again maybe this would be silly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571026818


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,26 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.

Review Comment:
   @pitrou 
   ```suggestion
   * The storage type of this extension is ``StringArray`` or
     or ``StringViewArray`` or ``StringViewArray``.
     Only UTF-8 encoded JSON is supported.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rouault (via GitHub)" <gi...@apache.org>.
rouault commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573296182


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   An interesting metadata that could be put is ``{"schema": { ... insert here JSON schema .... }}`` to express contraints on the payload



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1569052310


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   Perhaps mention that only UTF-8 encodings of JSON are supported.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061349391

   Right, the `media_type` information would enable end users to interpret how data is encoded. Alternatively we could have one extension per endoding (JSON, YAML, etc) or say that JSON is special and only provide it as a canonical type. I'm ok either way actually, just proposing a more general approach.
   
   > By the way, the notion of a "generic" extension type meant to represent arbitrary third-party data is similar to the ["other / unsupported data type"](https://lists.apache.org/thread/8d5ldl5cb7mms21rd15lhpfrv4j9no4n) currently being discussed on the ML. @lidavidm @westonpace
   > 
   > I don't see any reason to restrict such an "other" type to text data.
   
   Here we would limit ourselves to text based IANA registered media types.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592661795


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -51,7 +51,7 @@ types:
 
   3) Its serialization *must* be described in the proposal and should
      not require unduly implementation work or unusual software dependencies
-     (for example, a trivial custom text format or JSON would be acceptable).
+     (for example, a trivial custom text format or YAML would be acceptable).

Review Comment:
   @ianmcook is giving another example ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061348507

   I don't see how they're the same. One is about specific structured data and one is a way for a producer to signal unknown types without erroring the whole operation or dropping data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061355446

   JSON is not to consider as a _file format_ here but as a _data type_. Several database engines such as PostgreSQL allow for first-class JSON columns and that's what we're trying to convey (I don't think YAML is similar in this regard).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1570982701


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,24 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is an empty string.

Review Comment:
   Perhaps we could have a mechanism to encode JSON schema or similar? :thinking: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571516958


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,26 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.

Review Comment:
   Indeed. Thanks! :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1584195785


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   > 3\. Specialized metadata used internally (e.g. BigQuery might want to store what fields were columnarized, so we could reinstantiate it if the user exports data and then reimports it back).
   
   Note that for such internal metadata, you are always free that any other key to the field metadata of the column, and you don't necessarily have to add that to the value of the `ARROW:extension:metadata` key



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1589739730


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   > By this you mean serializing binary data into UTF-8 string and including it as a String? What would be a downside to allowing that?
   
   I think the argument was that we might want to support `BinaryArray` as a storage type since a JSON document could be encoded with a non-utf8 encoding and thus should not be stored in a `StringArray`.  However, I agree we probably don't want to worry about this since the RFC is pretty clear that JSON must be UTF-8 and users can always make up their own extension type if they need to.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592423422


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.

Review Comment:
   ```suggestion
   * Extension name: ``arrow.json``.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571022394


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   Fair enough!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1569070238


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   Removing and waiting for others to chime in.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573348702


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   > I would also argue that adding optional metadata in the future should be regarded as backwards compatible, even if we don't specify that here now
   
   I agree, but in practice what is backwards compatible depends on what implementations do. So I think it's worth being extra clear here that it should not be an error if there is unexpected metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "progger-dev (via GitHub)" <gi...@apache.org>.
progger-dev commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1575445365


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   I totally agree. I don't think we should canonicalize these non-standard extensions. I don't think producers should generate non-standard JSON. But I think consumers should be allowed to parse non-standard JSON. This is consistent with the spec in [RFC-8259](https://datatracker.ietf.org/doc/html/rfc8259#section-9).
   
   This also begs the question of what we do when JSON parsing fails.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rouault (via GitHub)" <gi...@apache.org>.
rouault commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1583444710


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   I would suggest that binary encodings of JSON should be advertized as a different extension, e.g. "arrow.bson"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "progger-dev (via GitHub)" <gi...@apache.org>.
progger-dev commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585322513


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   I think we should also add language that specifies what happens when you receive non-conformant JSON or see any kind of parse errors. I think it's important for components that sit in between producers and consumers (and thus act as both).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2098556532

   On line 54 of `CanonicalExtensions.rst`, JSON is cited as a hypothetical example of a canonical extension type:
   > for example, a trivial custom text format **or JSON** would be acceptable
   
    This should be removed or replaced since it is no longer hypothetical.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592675429


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: ``arrow.json``.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.

Review Comment:
   Let's use the same wording as in the spec, then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2100249185

   If there's no further comments I'll merge this today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2066401698

   If there's nothing further here I'll call a ML vote later today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061320269

   I'm sorry, but this doesn't seem to make sense to me. We already have a "generic text data format": it's the STRING type.
   
   What we're talking about is a JSON extension type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571026818


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,26 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.

Review Comment:
   @pitrou 
   ```suggestion
   * The storage type of this extension is ``StringArray`` or
     or ``StringViewArray`` or ``LargeStringViewArray``.
     Only UTF-8 encoded JSON is supported.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1571026818


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,26 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``StringViewArray``.

Review Comment:
   @pitrou 
   ```suggestion
   * The storage type of this extension is ``StringArray`` or
     or ``LargeStringArray`` or ``StringViewArray``.
     Only UTF-8 encoded JSON is supported.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061386906

   > JSON is not to consider as a _file format_ here but as a _data type_. Several database engines such as PostgreSQL allow for first-class JSON columns and that's what we're trying to convey (I don't think YAML is similar in this regard).
   
   That's a fair point for JSON being special.
   [Postgres does support XML](https://www.postgresql.org/docs/current/datatype-xml.html) and JSON colunms. but that seems a too small of a set for a generic type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1569047346


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   I think it's ok, but the Large variants can perhaps be omitted since we're unlikely to store a JSON value more than 2 GB large.
   
   Similarly, for `arrow.variable_shape_tensor` we're mandating a List type, not LargeList.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1569026700


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,33 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _text_data_format_extension:
+
+Text data format
+=================
+
+* Extension name: `arrow.text_data_format`.
+
+* The storage type of the extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.

Review Comment:
   @pitrou any thougts on storage type?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061524818

   Ok, can we reboot this into an actual JSON extension type?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592667968


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -51,7 +51,7 @@ types:
 
   3) Its serialization *must* be described in the proposal and should
      not require unduly implementation work or unusual software dependencies
-     (for example, a trivial custom text format or JSON would be acceptable).
+     (for example, a trivial custom text format or YAML would be acceptable).

Review Comment:
   ```suggestion
        (for example, a trivial custom text format or a JSON-based format would be acceptable).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592670297


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -51,7 +51,7 @@ types:
 
   3) Its serialization *must* be described in the proposal and should
      not require unduly implementation work or unusual software dependencies
-     (for example, a trivial custom text format or JSON would be acceptable).
+     (for example, a trivial custom text format or YAML would be acceptable).

Review Comment:
   Got it. Switched to @ianmcook's suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok merged PR #41257:
URL: https://github.com/apache/arrow/pull/41257


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573298088


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   @tustvold suggested metadata could hold JSON schema which would not be the case with UUID which could justify the difference between the types.
   I do agree with your reasoning that adding optional metadata later wouldn't be backwards incompatible, but perhaps we have users that need it now. @progger-dev any thoughts from you or someone on the bigquery team?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585212898


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   Does this make sense @emkornfield ?
   ```suggestion
     Only UTF-8 encoded JSON as specified in `rfc8259`_ is supported.
     Non-standard encodings using Binary types are not supported.
   ```
   
   > In the short term we should probably recommend that implementations should likely reject Binary annotated JSON fields
   
   By this you mean serializing binary data into UTF-8 string and including it as a String? Is there a downside to allowing it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585212898


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   Does this make sense @emkornfield ?
   ```suggestion
     Only UTF-8 encoded JSON is supported.
     Non-standard encodings using Binary types are not supported.
   ```
   
   > In the short term we should probably recommend that implementations should likely reject Binary annotated JSON fields
   
   By this you mean serializing binary data into UTF-8 string and including it as a String? So you see a downside to allowing that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585212898


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   Does this make sense @emkornfield ?
   ```suggestion
     Non-standard encodings using Binary types are not supported.
   ```
   
   > In the short term we should probably recommend that implementations should likely reject Binary annotated JSON fields
   
   By this you mean serializing binary data into UTF-8 string and including it as a String? So you see a downside to allowing that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1592685582


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: ``arrow.json``.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.

Review Comment:
   @jorisvandenbossche like so?
   ```suggestion
   * The storage type of this extension is ``String`` or
     or ``LargeString`` or ``StringView``.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2101900145

   After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 071ffaf2633eb58540a872514507ab362cc26fb4.
   
   None of the specified runs were found on the Conbench server.
   
   The [full Conbench report](https://github.com/apache/arrow/runs/24761129758) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1583465935


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   Sorry to clarify I was not talking about BSON, but rather UTF-16 or UTF-32 encodings for JSON (which one can debate if it is worth supporting based on RFC working about requiring UTF-8).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1583809602


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   Should we explicitly state in `CanonicalExtensions.rst` that this canonical extension type supports JSON _as defined in RFC-8259 (https://datatracker.ietf.org/doc/html/rfc8259)_?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1585212898


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.

Review Comment:
   Does this make sense @emkornfield ?
   ```suggestion
     Only UTF-8 encoded JSON is supported.
     Non-standard encodings using Binary types are not supported.
   ```
   
   > In the short term we should probably recommend that implementations should likely reject Binary annotated JSON fields
   
   By this you mean serializing binary data into UTF-8 string and including it as a String? What would be a downside to allowing that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "progger-dev (via GitHub)" <gi...@apache.org>.
progger-dev commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573871660


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   I had 3 things in my mind for what I would like to be able to use the metadata field for.
   
   1. Parsing options that describe what features were used to generate the JSON. E.g. trailing commas allowed, unquoted field names, etc.
   2. Schemas
   3. Specialized metadata used internally (e.g. BigQuery might want to store what fields were columnarized, so we could reinstantiate it if the user exports data and then reimports it back).
   
   None of these were requirements, but just some potential things that we might have used the metadata field for. I'm no longer at Google, so I'll tag @emkornfield for more recent thoughts from the BQ team.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "rouault (via GitHub)" <gi...@apache.org>.
rouault commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1573896963


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,27 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  or ``LargeStringArray`` or ``StringViewArray``.
+  Only UTF-8 encoded JSON is supported.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is either an empty string or a JSON string with an empty object.
+  In the future, additional fields may be added, but they are not required
+  to interpret the array.

Review Comment:
   > 1. Parsing options that describe what features were used to generate the JSON. E.g. trailing commas allowed, unquoted field names, etc
   
   ouch, I wouldn't call JSON with trailing commas, or unquoted field names JSON. Perhaps it would be wise to specify that JSON here is the one of https://www.json.org/json-en.html , that is ECMA-404 , otherwise that would defeat the purpose of interoperability between implementations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061267907

   :warning: GitHub issue #41256 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for JSON [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41257:
URL: https://github.com/apache/arrow/pull/41257#discussion_r1569050236


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -251,6 +251,25 @@ Variable shape tensor
    Values inside each **data** tensor element are stored in row-major/C-contiguous
    order according to the corresponding **shape**.
 
+.. _json_extension:
+
+JSON
+====
+
+* Extension name: `arrow.json`.
+
+* The storage type of this extension is ``StringArray`` or
+  ``LargeStringArray``or ``StringViewArray``.
+
+* Extension type parameters:
+
+  This type does not have any parameters.
+
+* Description of the serialization:
+
+  Metadata is an empty string.
+  - Example: ``""``

Review Comment:
   I don't think the example is useful ;-)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061352169

   In any case, I do think the framing here is wrong. Whether the format is "text" should be entirely irrelevant, and this is a distraction from the concrete aim of represent JSON data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-41256: [Format][Docs] Add a canonical extension type specification for a generic text data format (e.g. JSON) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41257:
URL: https://github.com/apache/arrow/pull/41257#issuecomment-2061350504

   @lidavidm See the PR submitted here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org