[arrow] branch master updated: ARROW-412: [Format][Documentation] Clarify that Buffer.size in Flatbuffers should reflect the actual memory size rather than the padded size

[we...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git


The following commit(s) were added to refs/heads/master by this push:
     new 26c56d2  ARROW-412: [Format][Documentation] Clarify that Buffer.size in Flatbuffers should reflect the actual memory size rather than the padded size
26c56d2 is described below

commit 26c56d21ff0fb19d111411a7a15b8e865b6ab3c1
Author: Wes McKinney <we...@apache.org>
AuthorDate: Sun Oct 6 16:43:54 2019 -0500

    ARROW-412: [Format][Documentation] Clarify that Buffer.size in Flatbuffers should reflect the actual memory size rather than the padded size
    
    There had previously been some questions about whether the `Buffer` Flatbuffers struct should include padding bytes in its `length` or not. This adds documentation that the "length" need not be prescriptive about this -- if the writer intends the reader to receive a padded Buffer, then the padding bytes can be included, but it is not a requirement. The protocol requires that padding bytes to a minimum multiple of 8 be written nonetheless.
    
    Closes #4370 from wesm/ARROW-412 and squashes the following commits:
    
    0da9d8304 <Wes McKinney> Clarify per mailing list discussion
    0b590559d <Wes McKinney> Add language about serialized Buffer metadata
    
    Authored-by: Wes McKinney <we...@apache.org>
    Signed-off-by: Wes McKinney <we...@apache.org>
---
 docs/source/format/Columnar.rst |  7 ++++++-
 format/Schema.fbs               | 11 +++++++----
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/docs/source/format/Columnar.rst b/docs/source/format/Columnar.rst
index 8411c2c..2ea007c 100644
--- a/docs/source/format/Columnar.rst
+++ b/docs/source/format/Columnar.rst
@@ -940,7 +940,12 @@ table above): ::
 
 The ``Buffer`` Flatbuffers value describes the location and size of a
 piece of memory. Generally these are interpreted relative to the
-**encapsulated message format** defined next.
+**encapsulated message format** defined below.
+
+The ``size`` field of ``Buffer`` is not required to account for padding
+bytes. Since this metadata can be used to communicate in-memory pointer
+addresses between libraries, it is recommended to set ``size`` to the actual
+memory size rather than the padded size.
 
 Byte Order (`Endianness`_)
 ---------------------------
diff --git a/format/Schema.fbs b/format/Schema.fbs
index 4ce66d6..206fb24 100644
--- a/format/Schema.fbs
+++ b/format/Schema.fbs
@@ -202,8 +202,8 @@ enum IntervalUnit: short { YEAR_MONTH, DAY_TIME}
 // A "calendar" interval which models types that don't necessarily
 // have a precise duration without the context of a base timestamp (e.g.
 // days can differ in length during day light savings time transitions).
-// YEAR_MONTH - Indicates the number of elapsed whole months, stored as 
-//   4-byte integers. 
+// YEAR_MONTH - Indicates the number of elapsed whole months, stored as
+//   4-byte integers.
 // DAY_TIME - Indicates the number of elapsed days and milliseconds,
 //   stored as 2 contiguous 32-bit integers (8-bytes in total).  Support
 //   of this IntervalUnit is not required for full arrow compatibility.
@@ -211,7 +211,7 @@ table Interval {
   unit: IntervalUnit;
 }
 
-// An absolute length of time unrelated to any calendar artifacts.  
+// An absolute length of time unrelated to any calendar artifacts.
 //
 // For the purposes of Arrow Implementations, adding this value to a Timestamp
 // ("t1") naively (i.e. simply summing the two number) is acceptable even
@@ -323,7 +323,10 @@ struct Buffer {
   offset: long;
 
   /// The absolute length (in bytes) of the memory buffer. The memory is found
-  /// from offset (inclusive) to offset + length (non-inclusive).
+  /// from offset (inclusive) to offset + length (non-inclusive). When building
+  /// messages using the encapsulated IPC message, padding bytes may be written
+  /// after a buffer, but such padding bytes do not need to be accounted for in
+  /// the size here.
   length: long;
 }