Re: [PR] GH-37876: [Format] Add list-view specification to arrow format [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #37877:
URL: https://github.com/apache/arrow/pull/37877#discussion_r1342657246


##########
docs/source/format/Columnar.rst:
##########
@@ -487,6 +499,102 @@ will be represented as follows: ::
           |-------------------------------|-----------------------|
           | 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 | unspecified (padding) |
 
+ListView Layout
+~~~~~~~~~~~~~~~
+
+The ListView layout is defined by three buffers: a validity bitmap, an offsets
+buffer, and an additional sizes buffer. Sizes and offsets have the identical bit
+width and both 32-bit and 64-bit signed integer options are supported.
+
+As in the List layout, the offsets encode the start position of each slot in the
+child array. In contrast to the List layout, list lengths are stored explicitly
+in the sizes buffer instead of inferred. This allows offsets to be out of order.
+Elements of the child array do not have to be stored in the same order they
+logically appear in the list elements of the parent array.
+
+When a value is null, the corresponding offset and size can have arbitrary
+values. When size is 0, the corresponding offset can have an arbitrary value.

Review Comment:
   Hmm... I wonder if we should be stricter here and accept only values that fall within the bounds of the child array. It might make life easier for language-strict implementations, and perhaps specific instrumentations such as ASan / UBsan.
   
   My question would be: is it detrimental in any way to be stricter? Intuitively, it doesn't seem performance would suffer (as populating a ListView array would typically keep a running offset, valid by construction).
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org