You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "erichanwang (via GitHub)" <gi...@apache.org> on 2026/08/10 00:14:49 UTC

[PR] GH-22342: [C++] [Documentation] Clarify Union type ids [arrow]

erichanwang opened a new pull request, #50839:
URL: https://github.com/apache/arrow/pull/50839

   ### Rationale for this change
   
   The Union layout documentation does not explain that the type IDs stored in the types buffer can differ from the child array indices, or how the optional `typeIds` metadata maps children to physical type IDs.
   
   ### What changes are included in this PR?
   
   - Document the relationship between Union child array indices, physical type IDs, and the optional `typeIds` metadata.
   - Clarify that the types buffer stores the type ID for each slot.
   
   Fixes #22342.
   
   ### Are these changes tested?
   
   `git diff --check` passes and the RST change was reviewed against `Schema.fbs` and the existing Union layout documentation. A local Sphinx/RST checker is not installed in this environment.
   
   ### Are there any user-facing changes?
   
   No.
   
   * GitHub Issue: #22342
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #50839:
URL: https://github.com/apache/arrow/pull/50839#discussion_r4040462584


##########
docs/source/format/Columnar.rst:
##########
@@ -864,6 +864,12 @@ A union is defined by an ordered sequence of types; each slot in the
 union can have a value chosen from these types. The types are named
 like a struct's fields, and the names are part of the type metadata.
 
+Each child type in a union has a type id (an 8-bit signed integer)
+that identifies it in the types buffer. By default, these type ids
+correspond directly to the child array indices (0, 1, ...), but an optional
+``typeIds`` metadata array can provide an explicit mapping from child array

Review Comment:
   It isn't clear from this how they actually do this, I think an example would make it clear.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-6012963925

   I'll wait for CI and then merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50839:
URL: https://github.com/apache/arrow/pull/50839#discussion_r4193410130


##########
docs/source/format/Columnar.rst:
##########
@@ -864,6 +864,12 @@ A union is defined by an ordered sequence of types; each slot in the
 union can have a value chosen from these types. The types are named
 like a struct's fields, and the names are part of the type metadata.
 
+Each child type in a union has a type id (an 8-bit signed integer)
+that identifies it in the types buffer. By default, these type ids
+correspond directly to the child array indices (0, 1, ...), but an optional
+``typeIds`` metadata array can provide an explicit mapping from child array
+offset to type id.

Review Comment:
   ```suggestion
   Each child type in a union has a logical type id (an 8-bit signed integer)
   that identifies it in the types buffer. By default, these type ids
   correspond directly to the child array indices (0, 1, ...), but an optional
   ``typeIds`` metadata array can provide an explicit mapping from child array
   index to type id.
   ```



##########
docs/source/format/Columnar.rst:
##########
@@ -878,10 +884,10 @@ Dense union represents a mixed-type array with 5 bytes of overhead for
 each value. Its physical layout is as follows:
 
 * One child array for each type
-* Types buffer: A buffer of 8-bit signed integers. Each type in the
-  union has a corresponding type id whose values are found in this
-  buffer. A union with more than 128 possible types can be modeled as
-  a union of unions.
+* Types buffer: A buffer of 8-bit signed integers, indicating the type
+  id of each slot. Note that these type ids are not necessarily the
+  same as the child array index (see above). A union with more than 128
+  possible types can be modeled as a union of unions.

Review Comment:
   ```suggestion
   * Types buffer: A buffer of 8-bit signed integers, indicating the logical type
     id of each slot. Note that these type ids are not necessarily the
     same as the child array index (see above). A union with more than 128
     possible types can be modeled as a union of unions.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #50839:
URL: https://github.com/apache/arrow/pull/50839#discussion_r4085510121


##########
docs/source/format/Columnar.rst:
##########
@@ -878,10 +884,10 @@ Dense union represents a mixed-type array with 5 bytes of overhead for
 each value. Its physical layout is as follows:
 
 * One child array for each type
-* Types buffer: A buffer of 8-bit signed integers. Each type in the
-  union has a corresponding type id whose values are found in this
-  buffer. A union with more than 128 possible types can be modeled as
-  a union of unions.
+* Types buffer: A buffer of 8-bit signed integers, indicating the type
+  id of each slot. Note that these type ids are not necessarily the
+  same as the child array index (see above). A union with more than 128

Review Comment:
   It would be nice to use "child array offset" here to be consistent with the text above (or change above to be "child array index")



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-5717718910

   :warning: GitHub issue #22342 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [C++] [Documentation] Clarify Union type ids [arrow]

Posted by "erichanwang (via GitHub)" <gi...@apache.org>.
erichanwang commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-5234601090

   Formatting follow-up:
   
   - Wrapped the new Union type-id paragraph to match the surrounding Arrow RST line-wrapping style.
   - `git diff --check` passes; no semantic content changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-5719717024

   I think the change makes sense, an example might clear up any remaining ambiguity


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [C++] [Documentation] Clarify Union type ids [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-5234592160

   :warning: GitHub issue #22342 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-5717726717

   @paleolimbot @tustvold @alamb What do you think about these clarifications in the format spec?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [C++] [Documentation] Clarify Union type ids [arrow]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-5244396244

   Nice focused improvement, LGTM!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #50839:
URL: https://github.com/apache/arrow/pull/50839#discussion_r4040889607


##########
docs/source/format/Columnar.rst:
##########
@@ -864,6 +864,12 @@ A union is defined by an ordered sequence of types; each slot in the
 union can have a value chosen from these types. The types are named
 like a struct's fields, and the names are part of the type metadata.
 
+Each child type in a union has a type id (an 8-bit signed integer)
+that identifies it in the types buffer. By default, these type ids
+correspond directly to the child array indices (0, 1, ...), but an optional
+``typeIds`` metadata array can provide an explicit mapping from child array

Review Comment:
   Also possibly calling it just typeIds metadata (the term "array" here I believe is talking about a different kind of array than it is later in the paragraph)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50839:
URL: https://github.com/apache/arrow/pull/50839#issuecomment-6012961606

   I've applied some minor modifications to address some of the comments, in the interest of moving this forward.
   
   I think adding an example can be done in a followup task if someone is motivated to do so.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-22342: [Format][Documentation] Clarify Union type ids [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #50839:
URL: https://github.com/apache/arrow/pull/50839


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org