You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Benjamin-Philip (via GitHub)" <gi...@apache.org> on 2026/07/14 13:16:30 UTC
[PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Benjamin-Philip opened a new pull request, #104:
URL: https://github.com/apache/arrow-erlang/pull/104
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## What issue does this PR close?
Closes #103.
## What's Changed
Inline documentation has been migrated to markdown.
Erlang/OTP, Rebar3 and `rebar3_ex_doc` have all been updated to the latest
stable version in order to use the new markdown support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#discussion_r3610749971
##########
src/arrow_variable_list_array.erl:
##########
@@ -15,30 +15,29 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides support for Arrow's Fixed-Size List Layout.
-%%
-%% This module provides support for the Fixed-Size List Layout[1], which is an
-%% layout that supports storing a list of lists of a specific length and
-%% nesting.
-%%
-%% == Invalid Input ==
-%%
-%% It is important that care is taken when passing input values to this module.
-%% For performance reasons, the input is not validated. The function crashes on
-%% nesting that is inconsitent: a. with the type, b. between elements. The
-%% behaviour on invalid input <strong>CANNOT BE GUARANTEED</strong>. Therefore,
-%% one must be careful to not to <strong>CRASH THE PROCESS</strong> or worse
-%% still, <strong>PRODUCE INVALID OUTPUT</strong>.
-%%
-%% Any input must follow the following rules:
-%%
-%% <ol>
-%% <li>The nesting of each element must be consistent with the type</li>
-%% <li>The nesting of each element must be consistent with each other</li>
-%% </ol>
-%%
-%% [1]: https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout
-module(arrow_variable_list_array).
+-moduledoc """
+Provides support for Arrow's Fixed-Size List Layout.
+
+This module provides support for the [Fixed-Size List
+Layout](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout),
+which is an layout that supports storing a list of lists of a specific length
+and nesting.
Review Comment:
The moduledoc describes this as “Fixed-Size List Layout”, but the module implements `variable_list` (and links to the variable-size list layout). This makes the rendered docs misleading.
##########
src/arrow_type.erl:
##########
@@ -15,123 +15,94 @@
% specific language governing permissions and limitations
% under the License.
-%%%-----------------------------------------------------------------------------
-%% @doc Provides representation for as well as functions for working with Arrow
-%% specific datatypes.
-%%
-%% This module provides a longhand (a 2 tuple) as well as a shorthand (an atom)
-%% represenations for representing a primitive types. Additionally it represents
-%% Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types
-%% that are supported by this package).
-%%
-%% Generally, when we say an array A has type X, we mean that each element in A
-%% has type X. Thus, an array of type of `Int8' would look like `[1, 2, 3]', but
-%% an array of type `List<Int8>' would look like `[[1, 2], [3]]'.
-%%
-%% == Primitive Types ==
-%%
-%% A Primitive Type is any type that refers to a single value (as opposed to a
-%% list of values). Currently it includes Booleans, Signed Integers, Unsigned
-%% Integers, Floats and Binaries. Both a longhand and shorthand syntax had been
-%% provided to initiate a type.
-%%
-%% The following is a comprehensive list of all supported primitive types:
-%%
-%% <ol>
-%% <li>Booleans</li>
-%% <li>Signed Integers: Includes `Int 8', `Int 16', `Int 32' and `Int 64'</li>
-%% <li>
-%% Unsigned Integers: Includes `UInt 8', `UInt 16', `UInt 32' and `UInt 64'
-%% </li>
-%% <li>Floats: Includes `Float 16', `Float 32' and `Float 64'</li>
-%% <li>Binaries</li>
-%% </ol>
-%%
-%% === Shorthand Syntax ===
-%%
-%% In the shorthand syntax, you can just use a single atom to refer to a type,
-%% such that the atom is `<name><size>'. The name is a single letter that refers
-%% to the type, and the size is that type's size in bits. Take the example of
-%% `Int 8'. Its name is `s', and size is `8'. Thus in shorthand, it is `s8'. In
-%% the case of datatypes without variations, such as Booleans and Binaries, they
-%% are represented by just their datatype. Do note that internally the longhand
-%% syntax is used.
-%%
-%% Here you can find the name of each type:
-%%
-%% <table border="1">
-%% <tr>
-%% <td>Type</td>
-%% <td>Name</td>
-%% <td>Example Atom</td>
-%% </tr>
-%% <tr>
-%% <td>Booleans</td>
-%% <td>N/A</td>
-%% <td>`bool'</td>
-%% </tr>
-%% <tr>
-%% <td>Signed Integer</td>
-%% <td>`s'</td>
-%% <td>`s64'</td>
-%% </tr>
-%% <tr>
-%% <td>Unsigned Integer</td>
-%% <td>`u'</td>
-%% <td>`u16'</td>
-%% </tr>
-%% <tr>
-%% <td>Float</td>
-%% <td>`f'</td>
-%% <td>`f32'</td>
-%% </tr>
-%% <tr>
-%% <td>Binaries</td>
-%% <td>N/A</td>
-%% <td>`bin'</td>
-%% </tr>
-%% </table>
-%%
-%% Types that are shorthand are under the `t:arrow_shorthand_type()' and
-%% `arrow_short_*()' types.
-%%
-%% === Longhand Syntax ===
-%%
-%% In longhand syntax, you use a 2 tuple to to refer to the type, such that the
-%% tuple is `{Name, Size}'. Thus, `Int 8' would be represented as `{s, 8}'. Do
-%% note that in the case or Booleans and Binaries their `Size' is `undefined', as
-%% the have variable or undefined size.
-%%
-%% Types that are longhand are under the `t:arrow_longhand_type()' and
-%% `arrow_long_*()' types.
-%%
-%% == Nested Type ==
-%%
-%% A Nested Type is any data structure that supports nesting. This can include
-%% Lists, FixedLists, Maps, Structs and more, but is curently limited to
-%% FixedLists. Nested Types are represented using a 3 tuple: `{NestedType, Type,
-%% Length}'. `NestedType' refers to the data structure that is nesting your
-%% primitive type. `Type' refers to the type that is being nested in the data
-%% structure. Do note that `Type' can be a Primitive Type, or another Nested
-%% Type. Both the shorthand and the longhand syntax can be used or Primitive
-%% Types.`Length' refers to the length of each element, and will only be
-%% required to represent FixedLists.
-%%
-%% It is important to remember while nesting that the type refers to the type of
-%% each element in an array. A list of `Int8' is 2 dimensional, and a list of
-%% `List<Int8>' is 3 dimensional.
-%%
-%% == Native Type ==
-%%
-%% A Native Type is any Erlang Datatype that can be serialized by `arrow',
-%% and is represented by the type `t:native_type()'. It currently only includes
-%% values that are primitive in nature such as booleans, integers, floats,
-%% binaries, and additionally the atoms `undefined' and `nil' to represent a
-%% `NULL' value.
-%%
-%% @end
-%%%-----------------------------------------------------------------------------
-module(arrow_type).
+-moduledoc """
+Provides representation for as well as functions for working with Arrow
+specific datatypes.
+
+This module provides a longhand (a 2 tuple) as well as a shorthand (an atom)
+represenations for representing a primitive types. Additionally it represents
+Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types
+that are supported by this package).
+
+Generally, when we say an array A has type X, we mean that each element in A
+has type X. Thus, an array of type of `Int8` would look like `[1, 2, 3]`, but
+an array of type `List<Int8>` would look like `[[1, 2], [3]]`.
+
+## Primitive Types
+
+A Primitive Type is any type that refers to a single value (as opposed to a
+list of values). Currently it includes Booleans, Signed Integers, Unsigned
+Integers, Floats and Binaries. Both a longhand and shorthand syntax had been
+provided to initiate a type.
+
+The following is a comprehensive list of all supported primitive types:
+
+1. Booleans
+2. Signed Integers: Includes `Int 8`, `Int 16`, `Int 32` and
+ `Int 64`
+3. Unsigned Integers: Includes `UInt 8`, `UInt 16`, `UInt 32` and
+ `UInt 64`
+4. Floats: Includes `Float 16`, `Float 32` and `Float 64`
+5. Binaries
+
+### Shorthand Syntax
+
+In the shorthand syntax, you can just use a single atom to refer to a type,
+such that the atom is `<name><size>`. The name is a single letter that refers
+to the type, and the size is that type's size in bits. Take the example of
+`Int 8`. Its name is `s`, and size is `8`. Thus in shorthand, it is `s8`. In
+the case of datatypes without variations, such as Booleans and Binaries, they
+are represented by just their datatype. Do note that internally the longhand
+syntax is used.
+
+Here you can find the name of each table:
+
+| Type | Name | Example Atom |
+|------------------|-------|--------------|
+| Booleans | N/A | `bool` |
+| Signed Integer | `s` | `s64` |
+| Unsigned Integer | `u ` | `u16 ` |
+| Float | `f ` | `f32 ` |
+| Binaries | N/A | `bin ` |
+
+Types that are shorthand are under the `t:arrow_shorthand_type()` and
+`arrow_short_*()` types.
+
+### Longhand Syntax
+
+In longhand syntax, you use a 2 tuple to to refer to the type, such that the
+tuple is `{Name, Size}`. Thus, `Int 8` would be represented as `{s, 8}`. Do
+note that in the case or Booleans and Binaries their `Size` is `undefined`, as
+the have variable or undefined size.
+
+Types that are longhand are under the `t:arrow_longhand_type()` and
+`arrow_long_*()` types.
+
+### Type
Review Comment:
The heading “### Type” introduces the Nested Type section; as-written it reads like a generic ‘type’ section and is inconsistent with the surrounding terminology (Primitive Types / Native Type).
##########
src/arrow_type.erl:
##########
@@ -15,123 +15,94 @@
% specific language governing permissions and limitations
% under the License.
-%%%-----------------------------------------------------------------------------
-%% @doc Provides representation for as well as functions for working with Arrow
-%% specific datatypes.
-%%
-%% This module provides a longhand (a 2 tuple) as well as a shorthand (an atom)
-%% represenations for representing a primitive types. Additionally it represents
-%% Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types
-%% that are supported by this package).
-%%
-%% Generally, when we say an array A has type X, we mean that each element in A
-%% has type X. Thus, an array of type of `Int8' would look like `[1, 2, 3]', but
-%% an array of type `List<Int8>' would look like `[[1, 2], [3]]'.
-%%
-%% == Primitive Types ==
-%%
-%% A Primitive Type is any type that refers to a single value (as opposed to a
-%% list of values). Currently it includes Booleans, Signed Integers, Unsigned
-%% Integers, Floats and Binaries. Both a longhand and shorthand syntax had been
-%% provided to initiate a type.
-%%
-%% The following is a comprehensive list of all supported primitive types:
-%%
-%% <ol>
-%% <li>Booleans</li>
-%% <li>Signed Integers: Includes `Int 8', `Int 16', `Int 32' and `Int 64'</li>
-%% <li>
-%% Unsigned Integers: Includes `UInt 8', `UInt 16', `UInt 32' and `UInt 64'
-%% </li>
-%% <li>Floats: Includes `Float 16', `Float 32' and `Float 64'</li>
-%% <li>Binaries</li>
-%% </ol>
-%%
-%% === Shorthand Syntax ===
-%%
-%% In the shorthand syntax, you can just use a single atom to refer to a type,
-%% such that the atom is `<name><size>'. The name is a single letter that refers
-%% to the type, and the size is that type's size in bits. Take the example of
-%% `Int 8'. Its name is `s', and size is `8'. Thus in shorthand, it is `s8'. In
-%% the case of datatypes without variations, such as Booleans and Binaries, they
-%% are represented by just their datatype. Do note that internally the longhand
-%% syntax is used.
-%%
-%% Here you can find the name of each type:
-%%
-%% <table border="1">
-%% <tr>
-%% <td>Type</td>
-%% <td>Name</td>
-%% <td>Example Atom</td>
-%% </tr>
-%% <tr>
-%% <td>Booleans</td>
-%% <td>N/A</td>
-%% <td>`bool'</td>
-%% </tr>
-%% <tr>
-%% <td>Signed Integer</td>
-%% <td>`s'</td>
-%% <td>`s64'</td>
-%% </tr>
-%% <tr>
-%% <td>Unsigned Integer</td>
-%% <td>`u'</td>
-%% <td>`u16'</td>
-%% </tr>
-%% <tr>
-%% <td>Float</td>
-%% <td>`f'</td>
-%% <td>`f32'</td>
-%% </tr>
-%% <tr>
-%% <td>Binaries</td>
-%% <td>N/A</td>
-%% <td>`bin'</td>
-%% </tr>
-%% </table>
-%%
-%% Types that are shorthand are under the `t:arrow_shorthand_type()' and
-%% `arrow_short_*()' types.
-%%
-%% === Longhand Syntax ===
-%%
-%% In longhand syntax, you use a 2 tuple to to refer to the type, such that the
-%% tuple is `{Name, Size}'. Thus, `Int 8' would be represented as `{s, 8}'. Do
-%% note that in the case or Booleans and Binaries their `Size' is `undefined', as
-%% the have variable or undefined size.
-%%
-%% Types that are longhand are under the `t:arrow_longhand_type()' and
-%% `arrow_long_*()' types.
-%%
-%% == Nested Type ==
-%%
-%% A Nested Type is any data structure that supports nesting. This can include
-%% Lists, FixedLists, Maps, Structs and more, but is curently limited to
-%% FixedLists. Nested Types are represented using a 3 tuple: `{NestedType, Type,
-%% Length}'. `NestedType' refers to the data structure that is nesting your
-%% primitive type. `Type' refers to the type that is being nested in the data
-%% structure. Do note that `Type' can be a Primitive Type, or another Nested
-%% Type. Both the shorthand and the longhand syntax can be used or Primitive
-%% Types.`Length' refers to the length of each element, and will only be
-%% required to represent FixedLists.
-%%
-%% It is important to remember while nesting that the type refers to the type of
-%% each element in an array. A list of `Int8' is 2 dimensional, and a list of
-%% `List<Int8>' is 3 dimensional.
-%%
-%% == Native Type ==
-%%
-%% A Native Type is any Erlang Datatype that can be serialized by `arrow',
-%% and is represented by the type `t:native_type()'. It currently only includes
-%% values that are primitive in nature such as booleans, integers, floats,
-%% binaries, and additionally the atoms `undefined' and `nil' to represent a
-%% `NULL' value.
-%%
-%% @end
-%%%-----------------------------------------------------------------------------
-module(arrow_type).
+-moduledoc """
+Provides representation for as well as functions for working with Arrow
+specific datatypes.
+
+This module provides a longhand (a 2 tuple) as well as a shorthand (an atom)
+represenations for representing a primitive types. Additionally it represents
+Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types
+that are supported by this package).
+
+Generally, when we say an array A has type X, we mean that each element in A
+has type X. Thus, an array of type of `Int8` would look like `[1, 2, 3]`, but
+an array of type `List<Int8>` would look like `[[1, 2], [3]]`.
+
+## Primitive Types
+
+A Primitive Type is any type that refers to a single value (as opposed to a
+list of values). Currently it includes Booleans, Signed Integers, Unsigned
+Integers, Floats and Binaries. Both a longhand and shorthand syntax had been
+provided to initiate a type.
+
+The following is a comprehensive list of all supported primitive types:
+
+1. Booleans
+2. Signed Integers: Includes `Int 8`, `Int 16`, `Int 32` and
+ `Int 64`
+3. Unsigned Integers: Includes `UInt 8`, `UInt 16`, `UInt 32` and
+ `UInt 64`
+4. Floats: Includes `Float 16`, `Float 32` and `Float 64`
+5. Binaries
+
+### Shorthand Syntax
+
+In the shorthand syntax, you can just use a single atom to refer to a type,
+such that the atom is `<name><size>`. The name is a single letter that refers
+to the type, and the size is that type's size in bits. Take the example of
+`Int 8`. Its name is `s`, and size is `8`. Thus in shorthand, it is `s8`. In
+the case of datatypes without variations, such as Booleans and Binaries, they
+are represented by just their datatype. Do note that internally the longhand
+syntax is used.
+
+Here you can find the name of each table:
+
+| Type | Name | Example Atom |
+|------------------|-------|--------------|
+| Booleans | N/A | `bool` |
+| Signed Integer | `s` | `s64` |
+| Unsigned Integer | `u ` | `u16 ` |
+| Float | `f ` | `f32 ` |
+| Binaries | N/A | `bin ` |
Review Comment:
The shorthand table includes trailing spaces inside backticks (e.g. `` `u ` ``, `` `u16 ` ``), which makes the documented atoms incorrect and will confuse users copying them.
##########
src/arrow_ipc_file.erl:
##########
@@ -15,27 +15,32 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides records and functions to deal with the IPC File.
-%%
-%% The IPC File[1] is an extension of the IPC Stream[2] that supports random
-%% access with the help of a footer which contains the offsets of all the
-%% messages.
-%%
-%% [1]: https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format
-%% [2]: https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format
-%% @end
-module(arrow_ipc_file).
+-moduledoc """
+Provides records and functions to deal with the IPC File. The [IPC
+File](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) is an
+extension of the [IPC Stream](
+https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) that
+supports random access with the help of a footer which contains the offsets of
+all the messages.
Review Comment:
The `IPC Stream` Markdown link is broken because the URL is split onto a new line inside the link parentheses, which will likely render as plain text instead of a hyperlink.
##########
src/arrow_buffer.erl:
##########
@@ -74,15 +71,16 @@ from_erlang(Data, Type) ->
end,
from_erlang(Data, Type, Len).
-%% @doc Creates a new buffer from a list of Erlang values or binaries, given its
-%% type and length
-%% @end
+-doc """
+Creates a new buffer from a list of Erlang values or binaries, given its
+and length
+""".
Review Comment:
The `from_erlang/3` docstring is missing the word “type” (“given its and length”), which reads like a formatting error in generated docs.
##########
src/arrow_buffer.erl:
##########
@@ -15,55 +15,52 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Buffer implementation for `arrow'.
-%% This module adds suppport for buffers, or Contiguous Memory Regions.
-%%
-%% There are multiple things to know about buffers[1]:
-%%
-%% <ol>
-%% <li>
-%% Each value it stores is called an element or a slot[2].
-%% </li>
-%% <li>
-%% Each slot's length (in bytes) is a positive integer. As a result when we
-%% say that a slot has a length of 1, we mean that each slot has a length of
-%% 1 byte.
-%% </li>
-%% <li>
-%% The buffer's length in the metadata refers to the unpadded binary's size in bytes.
-%% </li>
-%% <li>
-%% All buffers have a size that is a multiple of 64. If their data's length
-%% is not a multiple of 64, it must be padded (in this implementation, by
-%% zeros).
-%% </li>
-%% <li>
-%% Null values are represented in this implementation by zeros.
-%% </li>
-%% <li>
-%% In this implementation buffers can be initialized from raw bytes as data
-%% apart datatypes supported by Arrow. This is so that the Validity Bitmap
-%% Buffer can be initialized.
-%% </li>
-%% </ol>
-%%
-%% [1]: [https://arrow.apache.org/docs/format/Glossary.html#term-buffer]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Glossary.html#term-slot]
-%% @end
-module(arrow_buffer).
+
+%% TODO Check formatting with erlfmt
+%% erlfmt:ignore
+-moduledoc """
+Buffer implementation for `arrow`.
+
+This module adds suppport for buffers, or Contiguous Memory Regions. There are
Review Comment:
Spelling: “suppport” should be “support” in the moduledoc.
##########
src/arrow_offsets.erl:
##########
@@ -15,68 +15,66 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides support for Apache Arrow's Offsets
-%%
-%% Arrow has a concept of offsets[1] in order to tell the length of a slot[2] or
-%% a single element in an array of variable-size elements. This module provides
-%% support for generating offsets.
-%%
-%% There are couple of things to remember about offsets:
-%%
-%% <ol>
-%% <li>
-%% Each element in the offsets coresponds to the distance in bytes of the
-%% coresponding element in the values from the beginning of the buffer.
-%%
-%% I.E., distance of values[j] from beginning of the buffer = offsets[j].
-%% </li>
-%% <li>
-%% The very last element in the offsets is the length of the buffer, or
-%% distance of the end of the last slot from the beginning of the buffer.
-%% </li>
-%% <li>
-%% Thus, the offsets is one element longer than the values, or:
-%%
-%% length(offsets) == length(values) + 1
-%% </li>
-%% <li>
-%% Therefore, in order to find the length of a slot, we subtract the offset
-%% the current element from the offset of the next element, or:
-%%
-%% slot[j] = offsets[j + 1] - offsets[j]
-%% </li>
-%% <li>
-%% Null values have an offset of 0 as they take no memory in the buffer.
-%% Thus, the previous offset and the current offset are equivalent if the
-%% current element is a null.
-%% </li>
-%% </ol>
-%%
-%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#terminology]
-%% @end
-module(arrow_offsets).
--export([new/2, new/3]).
+-moduledoc """
+Provides support for Apache Arrow's Offsets
+
+Arrow has a concept of
+[offsets](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout)
+in order to tell the length of a
+[slot](https://arrow.apache.org/docs/format/Columnar.html#terminology) or a
+single element in an array of variable-size elements. This module provides
+support for generating offsets.
+
+There are couple of things to remember about offsets:
+
+Firstly, each element in the offsets coresponds to the distance in bytes of the
+coresponding element in the values from the beginning of the buffer.
Review Comment:
Spelling in moduledoc: “coresponds/coresponding” should be “corresponds/corresponding”.
##########
src/arrow_ipc_type.erl:
##########
@@ -90,7 +87,9 @@
%% from_erlang/1 %%
%%%%%%%%%%%%%%%%%%%
-%% @doc Returns the IPC Type for an `#array{}'.
+-doc """
+Returns the IPC Type for an `t:arrow_array:array()`.
+""".
Review Comment:
The doc comment refers to `t:arrow_array:array()`, which isn’t a valid Erlang type expression. Using `arrow_array:array()` keeps the docs copy/pasteable.
##########
src/arrow_ipc_schema.erl:
##########
@@ -15,58 +15,48 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with Schemas
-%%
-%% A Schema[1] represents a table, or a list of arrays of equal length. This
-%% module provides a record and a function to manage all the metadata required
-%% to represent a schema. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `endianness': The Endianness of the table. One of `little' or `big'.
-%% Defaults to `little'.
-%% </li>
-%% <li>
-%% `fields': The list of fields[2] in a table.
-%% </li>
-%% <li>
-%% `type': The Layout of the column
-%% </li>
-%% <li>
-%% `custom_metadata': A list of custom metadata in key-value format
-%% </li>
-%% <li>
-%% `features': Any features used by the table which may not be present in
-%% other implementations of Arrow.
-%% </li>
-%% </ol>
-%%
-%% Currently, big endianness, custom metadata and features are not supported,
-%% but they have been added for forwards comapatibility.
-%%
-%% You can find Schemas in the Arrow spec here[3].
-%%
-%% [1]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530]
-%%
-%% [2]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492]
-%%
-%% [3]: [https://arrow.apache.org/docs/format/Columnar.html#schema-message]
-%% @end
-module(arrow_ipc_schema).
+-moduledoc """
+Provides a record and functions to deal with Schemas A
+[Schema](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530)[1]
+represents a table, or a list of arrays of equal length. This module provides a
+record and a function to manage all the metadata required to represent a schema.
+Metadata such as:
+
+1. `endianness`: The Endianness of the table. One of `little` or
+ `big`. Defaults to `little`.
+2. `fields`: The list of [fields](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492) in a table.
+3. `type`: The Layout of the column
+4. `custom_metadata`: A list of custom metadata in key-value format
+5. `features`: Any features used by the table which may not be
+ present in other implementations of Arrow.
+
+Currently, big endianness, custom metadata and features are not supported, but
+they have been added for forwards comapatibility.
Review Comment:
Spelling in moduledoc: “comapatibility” should be “compatibility”.
##########
src/arrow_ipc_record_batch.erl:
##########
@@ -15,57 +15,47 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with RecordBatches
-%%
-%% A RecordBatch[1] represents a list of equal length arrays and their
-%% coresponding buffers. This module provides a record and a function to manage
-%% all the metadata required to represent a RecordBatch. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `length': The number of rows or records. In other words, the length of
-%% an array.
-%% </li>
-%% <li>
-%% `nodes': A list of maps, where each map has the length and null count
-%% of an array
-%% </li>
-%% <li>
-%% `buffers': A list of maps, where each map has the length and the offset
-%% (from the beginning of the message body) of a buffer of an array.
-%% </li>
-%% <li>
-%% `compression': The compression applied on the body of the Record Batch.
-%% Can either by `undefined' (i.e. no compression), `zstd' for Zstandard[2],
-%% or `lz4_frame' for LZ4 Frame[3]. Defaults to `undefined'.
-%% </li>
-%% </ol>
-%%
-%% Currently, compression is not supported, but it has been added for forwards
-%% comapatibility.
-%%
-%% You can find RecordBatches in the Arrow spec here[4].
-%%
-%% [1]: [https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102]
-%%
-%% [2]: [https://facebook.github.io/zstd/]
-%%
-%% [3]: [https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md]
-%%
-%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message]
-%% @end
-module(arrow_ipc_record_batch).
+-moduledoc """
+Provides a record and functions to deal with RecordBatches
+A [RecordBatch](https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102) represents a list of equal length arrays and their
+coresponding buffers. This module provides a record and a function to manage
+all the metadata required to represent a RecordBatch. Metadata such as:
Review Comment:
The moduledoc opening has a missing sentence break and a spelling error (“coresponding”), which reduces readability of generated docs.
##########
src/arrow_ipc_record_batch.erl:
##########
@@ -15,57 +15,47 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with RecordBatches
-%%
-%% A RecordBatch[1] represents a list of equal length arrays and their
-%% coresponding buffers. This module provides a record and a function to manage
-%% all the metadata required to represent a RecordBatch. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `length': The number of rows or records. In other words, the length of
-%% an array.
-%% </li>
-%% <li>
-%% `nodes': A list of maps, where each map has the length and null count
-%% of an array
-%% </li>
-%% <li>
-%% `buffers': A list of maps, where each map has the length and the offset
-%% (from the beginning of the message body) of a buffer of an array.
-%% </li>
-%% <li>
-%% `compression': The compression applied on the body of the Record Batch.
-%% Can either by `undefined' (i.e. no compression), `zstd' for Zstandard[2],
-%% or `lz4_frame' for LZ4 Frame[3]. Defaults to `undefined'.
-%% </li>
-%% </ol>
-%%
-%% Currently, compression is not supported, but it has been added for forwards
-%% comapatibility.
-%%
-%% You can find RecordBatches in the Arrow spec here[4].
-%%
-%% [1]: [https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102]
-%%
-%% [2]: [https://facebook.github.io/zstd/]
-%%
-%% [3]: [https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md]
-%%
-%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message]
-%% @end
-module(arrow_ipc_record_batch).
+-moduledoc """
+Provides a record and functions to deal with RecordBatches
+A [RecordBatch](https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102) represents a list of equal length arrays and their
+coresponding buffers. This module provides a record and a function to manage
+all the metadata required to represent a RecordBatch. Metadata such as:
+
+1. `length`: The number of rows or records. In other words, the
+ length of an array.
+2. `nodes`: A list of maps, where each map has the length and null
+ count of an array
+3. `buffers`: A list of maps, where each map has the length and the
+ offset (from the beginning of the message body) of a buffer of an
+ array.
+
+4. `compression`: The compression applied on the body of the Record
+ Batch. Can either by `undefined` (i.e. no compression), `zstd` for
+ [Zstandard](https://facebook.github.io/zstd/), or `lz4_frame` for [LZ4
+ Frame](https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md).
+ Defaults to `undefined`.
+
+Currently, compression is not supported, but it has been added for forwards
+comapatibility.
Review Comment:
Spelling in moduledoc: “comapatibility” should be “compatibility”.
##########
src/arrow_ipc_schema.erl:
##########
@@ -15,58 +15,48 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with Schemas
-%%
-%% A Schema[1] represents a table, or a list of arrays of equal length. This
-%% module provides a record and a function to manage all the metadata required
-%% to represent a schema. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `endianness': The Endianness of the table. One of `little' or `big'.
-%% Defaults to `little'.
-%% </li>
-%% <li>
-%% `fields': The list of fields[2] in a table.
-%% </li>
-%% <li>
-%% `type': The Layout of the column
-%% </li>
-%% <li>
-%% `custom_metadata': A list of custom metadata in key-value format
-%% </li>
-%% <li>
-%% `features': Any features used by the table which may not be present in
-%% other implementations of Arrow.
-%% </li>
-%% </ol>
-%%
-%% Currently, big endianness, custom metadata and features are not supported,
-%% but they have been added for forwards comapatibility.
-%%
-%% You can find Schemas in the Arrow spec here[3].
-%%
-%% [1]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530]
-%%
-%% [2]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492]
-%%
-%% [3]: [https://arrow.apache.org/docs/format/Columnar.html#schema-message]
-%% @end
-module(arrow_ipc_schema).
+-moduledoc """
+Provides a record and functions to deal with Schemas A
+[Schema](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530)[1]
+represents a table, or a list of arrays of equal length. This module provides a
+record and a function to manage all the metadata required to represent a schema.
Review Comment:
The moduledoc has a leftover footnote marker (`[1]`) and a missing sentence break (“Schemas A ...”), which hurts readability in generated docs.
##########
src/arrow_bitmap.erl:
##########
@@ -15,57 +15,49 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Validity Bitmap implementation for `arrow'.
-%%
-%% Defines a function `validity_bitmap/1' to return the Validity Bitmap[1] along
-%% with the Null Count[2], of an Array.
-%%
-%% An important thing to consider about our implementation of the Null Count is
-%% that we need to support both `undefined' and `nil' as null values as they are
-%% the conventions for null values in Erlang, and Elixir respectively.
-%%
-%% There are 5 important characteristics to remember about the validity bitmap:
-%%
-%% <ol>
-%% <li>
-%% A null value is represented by a 0 bit, and a non null value by a 1 bit.
-%% </li>
-%% <li>
-%% Every 8 elements's validities are batched into a byte, which are then
-%% reversed as Arrow uses least-significant bit (LSB) numbering (more in
-%% attached reference).
-%% </li>
-%% <li>
-%% If a "batch" consists of less than 8 elements, its validity needs to be
-%% padded by 0 bits so that it can make a byte.
-%% </li>
-%% <li>
-%% Each byte is stored in a slot of a Buffer (see docs for
-%% `arrow_buffer'). This buffer with the validities of each batch of 8
-%% elements make up what is called the Validity Bitmap.
-%% </li>
-%% <li>
-%% If the Null Count is 0, we can allocate the Validity Bitmap as a NULL
-%% pointer (which in Erlang's case is `undefined').
-%% </li>
-%% </ol>
-%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#null-count]
-%% @end
-module(arrow_bitmap).
--export([validity_bitmap/1]).
+-moduledoc """
+Validity Bitmap implementation for `arrow`.
+
+Defines a function `validity_bitmap/1` to return the [Validity
+Bitmap](https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps)
+along with the [Null
+Count](https://arrow.apache.org/docs/format/Columnar.html#null-count), of an
+Array.
+
+An important thing to consider about our implementation of the Null Count is
+that we need to support both `undefined` and `nil` as null values as they are
+the conventions for null values in Erlang, and Elixir respectively.
+
+There are 5 important characteristics to remember about the validity bitmap:
--include("arrow_buffer.hrl").
+1. A null value is represented by a 0 bit, and a non null value by a 1 bit.
+
+2. Every 8 elements's validities are batched into a byte, which are then
+ reversed as Arrow uses least-significant bit (LSB) numbering (more in
+ attached reference).
+
+3. If a "batch" consists of less than 8 elements, its validity needs to be
+ padded by 0 bits so that it can make a byte.
+
+4. Each byte is stored in a slot of a Buffer (see docs for `arrow_buffer`). This
+ buffer with the validities of each batch of 8 elements make up what is called
+ the Validity Bitmap.
+
+5. If the Null Count is 0, we can allocate the Validity Bitmap as a NULL pointer
+ (which in Erlang's case is `undefined`).
+""".
+-export([validity_bitmap/1]).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Validity Bitmap & Null Count %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%% @doc Returns the Validity Bitmap along with the Null Count, of
-%% an Array.
+-doc """
+Returns the Validity Bitmap along with the Null Count, of an Array.
+""".
-spec validity_bitmap(Value :: [arrow_type:native_type()] | list()) ->
- {Bitmap :: #buffer{}, non_neg_integer()}.
+ {Bitmap :: arrow_buffer:buffer(), non_neg_integer()}.
Review Comment:
`validity_bitmap/1` returns `{undefined, 0}` when there are no nulls, but the typespec currently states the bitmap is always `arrow_buffer:buffer()`. This is a spec/implementation mismatch (and will cause Dialyzer warnings).
##########
src/arrow_type.erl:
##########
@@ -15,123 +15,94 @@
% specific language governing permissions and limitations
% under the License.
-%%%-----------------------------------------------------------------------------
-%% @doc Provides representation for as well as functions for working with Arrow
-%% specific datatypes.
-%%
-%% This module provides a longhand (a 2 tuple) as well as a shorthand (an atom)
-%% represenations for representing a primitive types. Additionally it represents
-%% Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types
-%% that are supported by this package).
-%%
-%% Generally, when we say an array A has type X, we mean that each element in A
-%% has type X. Thus, an array of type of `Int8' would look like `[1, 2, 3]', but
-%% an array of type `List<Int8>' would look like `[[1, 2], [3]]'.
-%%
-%% == Primitive Types ==
-%%
-%% A Primitive Type is any type that refers to a single value (as opposed to a
-%% list of values). Currently it includes Booleans, Signed Integers, Unsigned
-%% Integers, Floats and Binaries. Both a longhand and shorthand syntax had been
-%% provided to initiate a type.
-%%
-%% The following is a comprehensive list of all supported primitive types:
-%%
-%% <ol>
-%% <li>Booleans</li>
-%% <li>Signed Integers: Includes `Int 8', `Int 16', `Int 32' and `Int 64'</li>
-%% <li>
-%% Unsigned Integers: Includes `UInt 8', `UInt 16', `UInt 32' and `UInt 64'
-%% </li>
-%% <li>Floats: Includes `Float 16', `Float 32' and `Float 64'</li>
-%% <li>Binaries</li>
-%% </ol>
-%%
-%% === Shorthand Syntax ===
-%%
-%% In the shorthand syntax, you can just use a single atom to refer to a type,
-%% such that the atom is `<name><size>'. The name is a single letter that refers
-%% to the type, and the size is that type's size in bits. Take the example of
-%% `Int 8'. Its name is `s', and size is `8'. Thus in shorthand, it is `s8'. In
-%% the case of datatypes without variations, such as Booleans and Binaries, they
-%% are represented by just their datatype. Do note that internally the longhand
-%% syntax is used.
-%%
-%% Here you can find the name of each type:
-%%
-%% <table border="1">
-%% <tr>
-%% <td>Type</td>
-%% <td>Name</td>
-%% <td>Example Atom</td>
-%% </tr>
-%% <tr>
-%% <td>Booleans</td>
-%% <td>N/A</td>
-%% <td>`bool'</td>
-%% </tr>
-%% <tr>
-%% <td>Signed Integer</td>
-%% <td>`s'</td>
-%% <td>`s64'</td>
-%% </tr>
-%% <tr>
-%% <td>Unsigned Integer</td>
-%% <td>`u'</td>
-%% <td>`u16'</td>
-%% </tr>
-%% <tr>
-%% <td>Float</td>
-%% <td>`f'</td>
-%% <td>`f32'</td>
-%% </tr>
-%% <tr>
-%% <td>Binaries</td>
-%% <td>N/A</td>
-%% <td>`bin'</td>
-%% </tr>
-%% </table>
-%%
-%% Types that are shorthand are under the `t:arrow_shorthand_type()' and
-%% `arrow_short_*()' types.
-%%
-%% === Longhand Syntax ===
-%%
-%% In longhand syntax, you use a 2 tuple to to refer to the type, such that the
-%% tuple is `{Name, Size}'. Thus, `Int 8' would be represented as `{s, 8}'. Do
-%% note that in the case or Booleans and Binaries their `Size' is `undefined', as
-%% the have variable or undefined size.
-%%
-%% Types that are longhand are under the `t:arrow_longhand_type()' and
-%% `arrow_long_*()' types.
-%%
-%% == Nested Type ==
-%%
-%% A Nested Type is any data structure that supports nesting. This can include
-%% Lists, FixedLists, Maps, Structs and more, but is curently limited to
-%% FixedLists. Nested Types are represented using a 3 tuple: `{NestedType, Type,
-%% Length}'. `NestedType' refers to the data structure that is nesting your
-%% primitive type. `Type' refers to the type that is being nested in the data
-%% structure. Do note that `Type' can be a Primitive Type, or another Nested
-%% Type. Both the shorthand and the longhand syntax can be used or Primitive
-%% Types.`Length' refers to the length of each element, and will only be
-%% required to represent FixedLists.
-%%
-%% It is important to remember while nesting that the type refers to the type of
-%% each element in an array. A list of `Int8' is 2 dimensional, and a list of
-%% `List<Int8>' is 3 dimensional.
-%%
-%% == Native Type ==
-%%
-%% A Native Type is any Erlang Datatype that can be serialized by `arrow',
-%% and is represented by the type `t:native_type()'. It currently only includes
-%% values that are primitive in nature such as booleans, integers, floats,
-%% binaries, and additionally the atoms `undefined' and `nil' to represent a
-%% `NULL' value.
-%%
-%% @end
-%%%-----------------------------------------------------------------------------
-module(arrow_type).
+-moduledoc """
+Provides representation for as well as functions for working with Arrow
+specific datatypes.
+
+This module provides a longhand (a 2 tuple) as well as a shorthand (an atom)
+represenations for representing a primitive types. Additionally it represents
+Nested Types (i.e. Lists) as well as Native Types (i.e. Erlang Native Types
+that are supported by this package).
Review Comment:
There are a couple of spelling/grammar errors in the moduledoc (“represenations”, “a primitive types”) that will show up in generated docs.
##########
src/arrow_variable_list_array.erl:
##########
@@ -15,30 +15,29 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides support for Arrow's Fixed-Size List Layout.
-%%
-%% This module provides support for the Fixed-Size List Layout[1], which is an
-%% layout that supports storing a list of lists of a specific length and
-%% nesting.
-%%
-%% == Invalid Input ==
-%%
-%% It is important that care is taken when passing input values to this module.
-%% For performance reasons, the input is not validated. The function crashes on
-%% nesting that is inconsitent: a. with the type, b. between elements. The
-%% behaviour on invalid input <strong>CANNOT BE GUARANTEED</strong>. Therefore,
-%% one must be careful to not to <strong>CRASH THE PROCESS</strong> or worse
-%% still, <strong>PRODUCE INVALID OUTPUT</strong>.
-%%
-%% Any input must follow the following rules:
-%%
-%% <ol>
-%% <li>The nesting of each element must be consistent with the type</li>
-%% <li>The nesting of each element must be consistent with each other</li>
-%% </ol>
-%%
-%% [1]: https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout
-module(arrow_variable_list_array).
+-moduledoc """
+Provides support for Arrow's Fixed-Size List Layout.
+
+This module provides support for the [Fixed-Size List
+Layout](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout),
+which is an layout that supports storing a list of lists of a specific length
+and nesting.
+
+## Invalid Input
+
+It is important that care is taken when passing input values to this
+module. For performance reasons, the input is not validated. The
+function crashes on nesting that is inconsitent: a. with the type, b.
Review Comment:
Spelling in moduledoc: “inconsitent” should be “inconsistent”.
##########
src/arrow_fixed_list_array.erl:
##########
@@ -15,50 +15,46 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides support for Arrow's Fixed-Size List Layout.
-%%
-%% This module provides support for the Fixed-Size List Layout[1], which is an
-%% layout that supports storing a list of lists of a specific length and
-%% nesting.
-%%
-%% == Invalid Input ==
-%%
-%% It is important that care is taken when passing input values to this module.
-%% For performance reasons, the input is not validated. The function crashes on
-%% nesting that is inconsitent: a. with the type, b. between elements. The lists
-%% are checked to have the same length as each other at the top level, but not
-%% for deeper levels. The behaviour of the module on invalid input
-%% <strong>CANNOT BE GUARANTEED</strong>. Therefore, one must be careful to not
-%% to <strong>CRASH THE PROCESS</strong> or worse still, <strong>PRODUCE INVALID
-%% OUTPUT</strong>.
-%%
-%% Any input must follow the following rules:
-%%
-%% <ol>
-%% <li>The length of each element must be consistent with the type</li>
-%% <li>The nesting of each element must be consistent with the type</li>
-%% <li>The length of each element must be consistent with each other</li>
-%% <li>The nesting of each element must be consistent with each other</li>
-%% <li>
-%% The nested type is a `fixed_list' (as only then can fixed size be
-%% guaranteed)
-%% </li>
-%% </ol>
-%%
-%% [1]: https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout
-module(arrow_fixed_list_array).
+-moduledoc """
+Provides support for Arrow's Fixed-Size List Layout.
+
+This module provides support for the [Fixed-Size List
+Layout](https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout),
+which is an layout that supports storing a list of lists of a specific length
+and nesting.
+
+## Invalid Input
+
+It is important that care is taken when passing input values to this module. For
+performance reasons, the input is not validated. The function crashes on nesting
+that is inconsitent: a. with the type, b. between elements. The lists are
Review Comment:
Spelling in moduledoc: “inconsitent” should be “inconsistent”.
##########
src/arrow_ipc_message.erl:
##########
@@ -15,73 +15,79 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with the Encapsulated Message
-%% Format.
-%%
-%% A Message is a serialized form of a Schema[1] or a RecordBatch[2] (which may
-%% be required to read a serialized array) along with some metadata. This module
-%% provides a record and a function to manage all the metadata required to
-%% represent a message. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `version': The Apache Arrow Format Version. One of v1..v5. Defaults to v5.
-%% </li>
-%% <li>
-%% `header': The metadata of the Schema or RecordBatch
-%% </li>
-%% <li>
-%% `body_length': The length of the body in bytes
-%% </li>
-%% <li>
-%% `custom_metadata': A list of custom metadata in key-value format
-%% </li>
-%% <li>
-%% `body': The actual body. Can be undefined (in the case of Schema) or a
-%% binary (in the case of Record Batch).
-%% </li>
-%% </ol>
-%%
-%% Currently, changing the version and custom metadata are not supported, but
-%% they have been added for forwards comapatibility.
-%%
-%% This module also provides the `to_ipc/1' function which serializes the
-%% message into the Encapsulated Message Format[3]. However, this function gives
-%% <em>incomplete output</em> with invalid metadata, because of an unsatisfied
-%% dependency on flatbuffers, which is required for serializing the metadata.
-%%
-%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#schema-message]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message]
-%%
-%% [3]: [https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format]
-%% @end
-module(arrow_ipc_message).
+-moduledoc """
+Provides a record and functions to deal with the Encapsulated Message
+Format.
+
+A Message is a serialized form of a
+[Schema](https://arrow.apache.org/docs/format/Columnar.html#schema-message) or a
+[RecordBatch](https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message)
+(which may be required to read a serialized array) along with some metadata.
+This module provides a record and a function to manage all the metadata required
+to represent a message. Metadata such as:
+
+1. `version`: The Apache Arrow Format Version. One of `v1..v5`.
+ Defaults to v5.
+2. `header`: The metadata of the Schema or RecordBatch
+3. `body_length`: The length of the body in bytes
+4. `custom_metadata`: A list of custom metadata in key-value format
+5. `body`: The actual body. Can be undefined (in the case of Schema)
+ or a binary (in the case of Record Batch).
+
+Currently, changing the version and custom metadata are not supported, but they
+have been added for forwards comapatibility.
Review Comment:
Spelling in moduledoc: “comapatibility” should be “compatibility”.
##########
src/arrow_ipc_field.erl:
##########
@@ -15,64 +15,64 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with individual fields in
-%% a Schema
-%%
-%% A Field[1] represents a single column in a table (which is represented by a
-%% Schema[2]). This module provides a record and a function to manage all the
-%% metadata required to represent a column. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `name': The Name of the column. This can either be a string or null.
-%% </li>
-%% <li>
-%% `nullable': A boolean representing whether a column can have null values.
-%% Generally true.
-%% </li>
-%% <li>
-%% `type': The Type of the column, and is of type
-%% `arrow_ipc_type:ipc_type()'
-%% </li>
-%% <li>
-%% `dictionary': A boolean representing if the column is dictionary encoded
-%% </li>
-%% <li>
-%% `children': The child fields of a nested datatype
-%% </li>
-%% <li>
-%% `custom_metadata': A list of custom metadata in key-value format
-%% </li>
-%% </ol>
-%%
-%% Currently, dictionary encoding and custom metadata are not supported, but
-%% they have been added for forwards comapatibility.
-%%
-%% [1]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492]
-%%
-%% [2]: [https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530]
-%% @end
-module(arrow_ipc_field).
+-moduledoc """
+Provides a record and functions to deal with individual fields in
+a Schema
+
+A
+[Field](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L469-L492)
+represents a single column in a table (which is represented by a
+[Schema](https://github.com/apache/arrow/blob/3456131ab7350bee5d9569ffd63d3f0ee713991c/format/Schema.fbs#L514-L530).
+This module provides a record and a function to manage all the metadata required
+to represent a column. Metadata such as:
+
+1. `name`: The Name of the column. This can either be a string or
+ null.
+2. `nullable`: A boolean representing whether a column can have null
+ values. Generally true.
+3. `type`: The Type of the column, and is of type
+ `t:arrow_ipc_type:ipc_type/0`
+4. `dictionary`: A boolean representing if the column is dictionary
+ encoded
+5. `children`: The child fields of a nested datatype
+6. `custom_metadata`: A list of custom metadata in key-value format
+
+Currently, dictionary encoding and custom metadata are not supported, but
+they have been added for forwards comapatibility.
Review Comment:
Spelling in moduledoc: “comapatibility” should be “compatibility”.
##########
src/arrow_ipc_record_batch.erl:
##########
@@ -15,57 +15,47 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record and functions to deal with RecordBatches
-%%
-%% A RecordBatch[1] represents a list of equal length arrays and their
-%% coresponding buffers. This module provides a record and a function to manage
-%% all the metadata required to represent a RecordBatch. Metadata such as:
-%%
-%% <ol>
-%% <li>
-%% `length': The number of rows or records. In other words, the length of
-%% an array.
-%% </li>
-%% <li>
-%% `nodes': A list of maps, where each map has the length and null count
-%% of an array
-%% </li>
-%% <li>
-%% `buffers': A list of maps, where each map has the length and the offset
-%% (from the beginning of the message body) of a buffer of an array.
-%% </li>
-%% <li>
-%% `compression': The compression applied on the body of the Record Batch.
-%% Can either by `undefined' (i.e. no compression), `zstd' for Zstandard[2],
-%% or `lz4_frame' for LZ4 Frame[3]. Defaults to `undefined'.
-%% </li>
-%% </ol>
-%%
-%% Currently, compression is not supported, but it has been added for forwards
-%% comapatibility.
-%%
-%% You can find RecordBatches in the Arrow spec here[4].
-%%
-%% [1]: [https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102]
-%%
-%% [2]: [https://facebook.github.io/zstd/]
-%%
-%% [3]: [https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md]
-%%
-%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message]
-%% @end
-module(arrow_ipc_record_batch).
+-moduledoc """
+Provides a record and functions to deal with RecordBatches
+A [RecordBatch](https://github.com/apache/arrow/blob/16328f0ccc73b7df665b4a18feb6adf26b7aa0e2/format/Message.fbs#L81-L102) represents a list of equal length arrays and their
+coresponding buffers. This module provides a record and a function to manage
+all the metadata required to represent a RecordBatch. Metadata such as:
+
+1. `length`: The number of rows or records. In other words, the
+ length of an array.
+2. `nodes`: A list of maps, where each map has the length and null
+ count of an array
+3. `buffers`: A list of maps, where each map has the length and the
+ offset (from the beginning of the message body) of a buffer of an
+ array.
+
+4. `compression`: The compression applied on the body of the Record
+ Batch. Can either by `undefined` (i.e. no compression), `zstd` for
Review Comment:
Grammar in moduledoc: “Can either by” should be “Can either be”.
##########
src/arrow_fixed_list_array.erl:
##########
@@ -15,50 +15,46 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides support for Arrow's Fixed-Size List Layout.
-%%
-%% This module provides support for the Fixed-Size List Layout[1], which is an
-%% layout that supports storing a list of lists of a specific length and
-%% nesting.
-%%
-%% == Invalid Input ==
-%%
-%% It is important that care is taken when passing input values to this module.
-%% For performance reasons, the input is not validated. The function crashes on
-%% nesting that is inconsitent: a. with the type, b. between elements. The lists
-%% are checked to have the same length as each other at the top level, but not
-%% for deeper levels. The behaviour of the module on invalid input
-%% <strong>CANNOT BE GUARANTEED</strong>. Therefore, one must be careful to not
-%% to <strong>CRASH THE PROCESS</strong> or worse still, <strong>PRODUCE INVALID
-%% OUTPUT</strong>.
-%%
-%% Any input must follow the following rules:
-%%
-%% <ol>
-%% <li>The length of each element must be consistent with the type</li>
-%% <li>The nesting of each element must be consistent with the type</li>
-%% <li>The length of each element must be consistent with each other</li>
-%% <li>The nesting of each element must be consistent with each other</li>
-%% <li>
-%% The nested type is a `fixed_list' (as only then can fixed size be
-%% guaranteed)
-%% </li>
-%% </ol>
-%%
-%% [1]: https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout
-module(arrow_fixed_list_array).
+-moduledoc """
+Provides support for Arrow's Fixed-Size List Layout.
+
+This module provides support for the [Fixed-Size List
+Layout](https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout),
+which is an layout that supports storing a list of lists of a specific length
Review Comment:
Grammar in moduledoc: “an layout” should be “a layout”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#discussion_r3579315505
##########
src/arrow_array.erl:
##########
@@ -148,42 +116,42 @@ from_erlang(Layout, Value, Opts) ->
%% Array Data and Metadata Access %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%% @doc Returns the layout of an array.
+-doc "Returns the layout of an array.".
-spec layout(Array :: #array{}) -> Layout :: layout().
layout(Array) ->
Array#array.layout.
-%% @doc Returns the type of an array.
+-doc "Returns the type of an array.".
-spec type(Array :: #array{}) -> Type :: arrow_type:arrow_type().
type(Array) ->
Array#array.type.
-%% @doc Returns the length of an array.
+-doc "Returns the length of an array.".
-spec len(Array :: #array{}) -> Length :: pos_integer().
len(Array) ->
Array#array.len.
-%% @doc Returns the length of an array.
+-doc "Returns the length of an array.".
Review Comment:
`element_len/1`’s doc string currently repeats `len/1` (“Returns the length of an array.”). Since `element_len` is the per-element length, the doc should say that explicitly.
##########
src/arrow_array.erl:
##########
@@ -15,87 +15,51 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record, a behaviour, and functions to work with Apache Arrow
-%% Arrays.
-%%
-%% This module as mentioned, provides conveniences for working with Apache Arrow
-%% Arrays[1] of different Layouts[2]. Firstly, it provides a record to represent
-%% all the Layout. Secondly, it provides a behaviour for different layouts to
-%% adhere to common functionality. Lastly, it provides functions to work with
-%% Arrays in a common manner.
-%%
-%% == The Structure of an Array ==
-%%
-%% `arrow''s implementation of an Array has the following fields in its
-%% record definition:
-%%
-%% <ol>
-%% <li>
-%% `layout', of type {@link atom()}, which represents the Layout of the
-%% Array.
-%% </li>
-%% <li>
-%% `type', of type {@link arrow_type:arrow_type()}, which represents
-%% the Logical Type[3] of the Array.
-%% </li>
-%% <li>`len', of type {@link pos_integer()}, which represents the Array's Length[4].</li>
-%% <li>
-%% `element_len', of type {@link pos_integer()} or `undefined', which
-%% represents the Length of each element in an Array. Currently it only has
-%% an integer value in the Fixed-Size List Layout[5]
-%% </li>
-%% <li>
-%% `null_count', of type {@link non_neg_integer()}, which represents the
-%% Array's Null Count[4], or the number of undefined values in the Array.
-%% </li>
-%% <li>
-%% `validity_bitmap', which is a buffer (`arrow_buffer') or the atom
-%% `undefined', which represents the Array's Validity Bitmap[6].
-%% </li>
-%% <li>
-%% `offsets', which is a buffer (`arrow_buffer') which represents the
-%% Offsets[7], or the start position of each slot in the data buffer of an
-%% Array.
-%% </li>
-%% <li>
-%% `data', which is a buffer (`arrow_buffer'), which represents the
-%% Array's Value Buffer, whose layout differs based on the Array Layout.
-%% </li>
-%% </ol>
-%%
-%% Certain fields are not required for certain layouts. For example, for the
-%% Fixed-Sized Primitive Layout, the offsets field is not required, in which
-%% case it is assigned as `undefined'. Similarly, the `validity_bitmap' is not
-%% required if there are no null values, in which case it is also assigned as
-%% `undefined'.
-%%
-%% == The Behaviour of an Array ==
-%%
-%% As of right now, a layout needs to implement the `c:from_erlang/2' callback. This is
-%% then used in the `from_erlang/3' function to create new arrays.
-%%
-%% == Functions for working with Arrays ==
-%%
-%% As of right now, only functions to access the various fields, to create
-%% new arrays, and to serialize arrays to arrow exist.
-%%
-%% == References ==
-%%
-%% [1]: [https://arrow.apache.org/docs/format/Glossary.html#term-array]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Glossary.html#term-physical-layout]
-%%
-%% [3]: [https://arrow.apache.org/docs/format/Glossary.html#term-type]
-%%
-%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#null-count]
-%%
-%% [5]: [https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout]
-%%
-%% [6]: [https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps]
-%%
-%% [7]: [https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout]
-%% @end
-module(arrow_array).
+-moduledoc """
+Provides a record, a behaviour, and functions to work with Apache Arrow Arrays.
+
+This module as mentioned, provides conveniences for working with Apache Arrow
+[Arrays](https://arrow.apache.org/docs/format/Glossary.html#term-array) of
+different
+[Layouts](https://arrow.apache.org/docs/format/Glossary.html#term-physical-layout).
+Firstly, it provides a record to represent all the Layout. Secondly, it provides
+a behaviour for different layouts to adhere to common functionality. Lastly, it
+provides functions to work with Arrays in a common manner.
+
+# The Structure of an Array
+
+`arrow`'s implementation of an Array has the following fields in its record
+definition, each corresponding to a field in the Arrow Columnar Format
+specification:
+
+| Field | Type | Representing |
+|-------------------|-----------------------------|-------------------------------------------------------------------------------------------|
+| `layout` | `t:atom/0` | Layout |
+| `type` | `t:arrow_type:arrow_type/0` | [Logical Type](https://arrow.apache.org/docs/format/Glossary.html#term-type) |
+| `len` | `t:pos_integer/0` | [Length](https://arrow.apache.org/docs/format/Columnar.html#array-lengths) |
+| `element_len` | `t:pos_integer/0` | Length of each element |
+| `null_count` | `t:non_neg_integer/0` | [Null Count](https://arrow.apache.org/docs/format/Columnar.html#null-count) |
+| `validity_bitmap` | `arrow_buffer` | [Validity Bitmap](https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps) |
+| `offsets` | `arrow_buffer` | [Offsets](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) |
+| `data` | `arrow_buffer` | Value Buffer |
Review Comment:
The moduledoc table’s type column doesn’t match the actual `#array{}` record fields (see `src/arrow_array.hrl`): `element_len` can be `undefined`, `validity_bitmap`/`offsets` are `#buffer{} | undefined`, and `data` can be a nested `#array{}`. This makes the rendered docs misleading.
##########
src/arrow_array.erl:
##########
@@ -192,24 +160,24 @@ data(Array) ->
%% Array Serialization %%
%%%%%%%%%%%%%%%%%%%%%%%%%
-%% @doc Serializes an array into the Arrow binary form.
-%%
-%% Serializes the buffers of an Array and concatenates them in the following
-%% order:
-%%
-%% <ol>
-%% <li>`validity'</li>
-%% <li>`offsets'</li>
-%% <li>`data'</li>
-%% </ol>
-%%
-%% In case an array doesn't have any of the following buffers, it is ommitted.
-%% (e.g. validity in arrays with a null count of 0, offsets in fixed primitive
-%% arrays). In the case of a nested array, `data' will be serialized form of
-%% nested array.
-%%
-%% Do note that this is just binary form that includes the buffers in an Array,
-%% and not IPC.
+-doc """
+Serializes an array into the Arrow binary form.
+
+Serializes the buffers of an Array and concatenates them in the following
+order:
+
+1. `validity`
+2. `offsets`
+3. `data`
+
+In case an array doesn't have any of the following buffers, it is ommitted.
+(e.g. validity in arrays with a null count of 0, offsets in fixed primitive
+arrays). In the case of a nested array, `data` will be serialized form of nested
+array.
+
+Do note that this is just binary form that includes the buffers in an Array,
+and not IPC.
+""".
Review Comment:
This doc block has a typo (“ommitted”) and a few grammar issues that will show up in the generated Markdown output. Also, there’s a trailing space after the opening `"""`.
##########
.github/workflows/erlang.yml:
##########
@@ -81,8 +81,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
- otp-version: 25.1.0
- rebar3-version: '3.18.0'
+ otp-version: 29.0
+ rebar3-version: 3.27.0
Review Comment:
Same YAML version-coercion risk here: unquoted `29.0` can be treated as a number and lose the `.0` when converted to a string for action inputs.
##########
.github/workflows/erlang.yml:
##########
@@ -43,8 +43,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
- otp-version: 25.1.0
- rebar3-version: '3.18.0'
+ otp-version: 29.0
+ rebar3-version: 3.27.0
Review Comment:
`otp-version: 29.0` will be parsed by YAML as a number (unlike the old `25.1.0`, which is a string), and may get stringified as `"29"` when passed to the action. Quoting avoids accidental version coercion and keeps behavior consistent.
##########
.github/workflows/erlang.yml:
##########
@@ -122,8 +122,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
- otp-version: 25.1.0
- rebar3-version: '3.18.0'
+ otp-version: 29.0
+ rebar3-version: 3.27.0
Review Comment:
Same YAML version-coercion risk here: unquoted `29.0` can be treated as a number and lose the `.0` when converted to a string for action inputs.
##########
.github/workflows/docs.yml:
##########
@@ -51,8 +51,8 @@ jobs:
- name: Install Erlang/OTP
uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1.24.1
with:
- otp-version: 25.1.0
- rebar3-version: '3.18.0'
+ otp-version: 29.0
+ rebar3-version: 3.27.0
Review Comment:
Same YAML version-coercion risk here: unquoted `29.0` can be treated as a number and lose the `.0` when converted to a string for action inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Benjamin-Philip (via GitHub)" <gi...@apache.org>.
Benjamin-Philip commented on code in PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#discussion_r3580459567
##########
src/arrow_array.erl:
##########
@@ -15,87 +15,51 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides a record, a behaviour, and functions to work with Apache Arrow
-%% Arrays.
-%%
-%% This module as mentioned, provides conveniences for working with Apache Arrow
-%% Arrays[1] of different Layouts[2]. Firstly, it provides a record to represent
-%% all the Layout. Secondly, it provides a behaviour for different layouts to
-%% adhere to common functionality. Lastly, it provides functions to work with
-%% Arrays in a common manner.
-%%
-%% == The Structure of an Array ==
-%%
-%% `arrow''s implementation of an Array has the following fields in its
-%% record definition:
-%%
-%% <ol>
-%% <li>
-%% `layout', of type {@link atom()}, which represents the Layout of the
-%% Array.
-%% </li>
-%% <li>
-%% `type', of type {@link arrow_type:arrow_type()}, which represents
-%% the Logical Type[3] of the Array.
-%% </li>
-%% <li>`len', of type {@link pos_integer()}, which represents the Array's Length[4].</li>
-%% <li>
-%% `element_len', of type {@link pos_integer()} or `undefined', which
-%% represents the Length of each element in an Array. Currently it only has
-%% an integer value in the Fixed-Size List Layout[5]
-%% </li>
-%% <li>
-%% `null_count', of type {@link non_neg_integer()}, which represents the
-%% Array's Null Count[4], or the number of undefined values in the Array.
-%% </li>
-%% <li>
-%% `validity_bitmap', which is a buffer (`arrow_buffer') or the atom
-%% `undefined', which represents the Array's Validity Bitmap[6].
-%% </li>
-%% <li>
-%% `offsets', which is a buffer (`arrow_buffer') which represents the
-%% Offsets[7], or the start position of each slot in the data buffer of an
-%% Array.
-%% </li>
-%% <li>
-%% `data', which is a buffer (`arrow_buffer'), which represents the
-%% Array's Value Buffer, whose layout differs based on the Array Layout.
-%% </li>
-%% </ol>
-%%
-%% Certain fields are not required for certain layouts. For example, for the
-%% Fixed-Sized Primitive Layout, the offsets field is not required, in which
-%% case it is assigned as `undefined'. Similarly, the `validity_bitmap' is not
-%% required if there are no null values, in which case it is also assigned as
-%% `undefined'.
-%%
-%% == The Behaviour of an Array ==
-%%
-%% As of right now, a layout needs to implement the `c:from_erlang/2' callback. This is
-%% then used in the `from_erlang/3' function to create new arrays.
-%%
-%% == Functions for working with Arrays ==
-%%
-%% As of right now, only functions to access the various fields, to create
-%% new arrays, and to serialize arrays to arrow exist.
-%%
-%% == References ==
-%%
-%% [1]: [https://arrow.apache.org/docs/format/Glossary.html#term-array]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Glossary.html#term-physical-layout]
-%%
-%% [3]: [https://arrow.apache.org/docs/format/Glossary.html#term-type]
-%%
-%% [4]: [https://arrow.apache.org/docs/format/Columnar.html#null-count]
-%%
-%% [5]: [https://arrow.apache.org/docs/format/Columnar.html#fixed-size-list-layout]
-%%
-%% [6]: [https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps]
-%%
-%% [7]: [https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout]
-%% @end
-module(arrow_array).
+-moduledoc """
+Provides a record, a behaviour, and functions to work with Apache Arrow Arrays.
+
+This module as mentioned, provides conveniences for working with Apache Arrow
+[Arrays](https://arrow.apache.org/docs/format/Glossary.html#term-array) of
+different
+[Layouts](https://arrow.apache.org/docs/format/Glossary.html#term-physical-layout).
+Firstly, it provides a record to represent all the Layout. Secondly, it provides
+a behaviour for different layouts to adhere to common functionality. Lastly, it
+provides functions to work with Arrays in a common manner.
+
+# The Structure of an Array
+
+`arrow`'s implementation of an Array has the following fields in its record
+definition, each corresponding to a field in the Arrow Columnar Format
+specification:
+
+| Field | Type | Representing |
+|-------------------|-----------------------------|-------------------------------------------------------------------------------------------|
+| `layout` | `t:atom/0` | Layout |
+| `type` | `t:arrow_type:arrow_type/0` | [Logical Type](https://arrow.apache.org/docs/format/Glossary.html#term-type) |
+| `len` | `t:pos_integer/0` | [Length](https://arrow.apache.org/docs/format/Columnar.html#array-lengths) |
+| `element_len` | `t:pos_integer/0` | Length of each element |
+| `null_count` | `t:non_neg_integer/0` | [Null Count](https://arrow.apache.org/docs/format/Columnar.html#null-count) |
+| `validity_bitmap` | `arrow_buffer` | [Validity Bitmap](https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps) |
+| `offsets` | `arrow_buffer` | [Offsets](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) |
+| `data` | `arrow_buffer` | Value Buffer |
Review Comment:
I address this in the following paragraph in order to keep the main table simple. Non-issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Benjamin-Philip (via GitHub)" <gi...@apache.org>.
Benjamin-Philip commented on PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#issuecomment-5029556212
Rebased onto main. This is a PR where further Copilot reviews would have been helpful. It's been fairly good at catching silly spelling and grammar mistakes.
Anyway, you can have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Benjamin-Philip (via GitHub)" <gi...@apache.org>.
Benjamin-Philip commented on PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#issuecomment-5023988934
@kou, I'll rebase on main once you merge #108. Hopefully we should have the CI
working again (and the docs re-deployed for this PR instead of #108) after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#issuecomment-4969596259
Preview URL: https://Benjamin-Philip.github.io/arrow-erlang-1
If the preview URL doesn't work, you may have forgotten to configure your fork repository for preview.
See https://github.com/apache/arrow-erlang/blob/main/CONTRIBUTING.md#forks for instructions on how to configure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Benjamin-Philip (via GitHub)" <gi...@apache.org>.
Benjamin-Philip commented on code in PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#discussion_r3614275656
##########
src/arrow_ipc_file.erl:
##########
@@ -15,27 +15,32 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Provides records and functions to deal with the IPC File.
-%%
-%% The IPC File[1] is an extension of the IPC Stream[2] that supports random
-%% access with the help of a footer which contains the offsets of all the
-%% messages.
-%%
-%% [1]: https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format
-%% [2]: https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format
-%% @end
-module(arrow_ipc_file).
+-moduledoc """
+Provides records and functions to deal with the IPC File. The [IPC
+File](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) is an
+extension of the [IPC Stream](
+https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) that
+supports random access with the help of a footer which contains the offsets of
+all the messages.
Review Comment:
Non issue, it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore: Migrate docs from EDoc to Markdown [arrow-erlang]
Posted by "Benjamin-Philip (via GitHub)" <gi...@apache.org>.
Benjamin-Philip commented on code in PR #104:
URL: https://github.com/apache/arrow-erlang/pull/104#discussion_r3614348158
##########
src/arrow_bitmap.erl:
##########
@@ -15,57 +15,49 @@
% specific language governing permissions and limitations
% under the License.
-%% @doc Validity Bitmap implementation for `arrow'.
-%%
-%% Defines a function `validity_bitmap/1' to return the Validity Bitmap[1] along
-%% with the Null Count[2], of an Array.
-%%
-%% An important thing to consider about our implementation of the Null Count is
-%% that we need to support both `undefined' and `nil' as null values as they are
-%% the conventions for null values in Erlang, and Elixir respectively.
-%%
-%% There are 5 important characteristics to remember about the validity bitmap:
-%%
-%% <ol>
-%% <li>
-%% A null value is represented by a 0 bit, and a non null value by a 1 bit.
-%% </li>
-%% <li>
-%% Every 8 elements's validities are batched into a byte, which are then
-%% reversed as Arrow uses least-significant bit (LSB) numbering (more in
-%% attached reference).
-%% </li>
-%% <li>
-%% If a "batch" consists of less than 8 elements, its validity needs to be
-%% padded by 0 bits so that it can make a byte.
-%% </li>
-%% <li>
-%% Each byte is stored in a slot of a Buffer (see docs for
-%% `arrow_buffer'). This buffer with the validities of each batch of 8
-%% elements make up what is called the Validity Bitmap.
-%% </li>
-%% <li>
-%% If the Null Count is 0, we can allocate the Validity Bitmap as a NULL
-%% pointer (which in Erlang's case is `undefined').
-%% </li>
-%% </ol>
-%% [1]: [https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps]
-%%
-%% [2]: [https://arrow.apache.org/docs/format/Columnar.html#null-count]
-%% @end
-module(arrow_bitmap).
--export([validity_bitmap/1]).
+-moduledoc """
+Validity Bitmap implementation for `arrow`.
+
+Defines a function `validity_bitmap/1` to return the [Validity
+Bitmap](https://arrow.apache.org/docs/format/Columnar.html#validity-bitmaps)
+along with the [Null
+Count](https://arrow.apache.org/docs/format/Columnar.html#null-count), of an
+Array.
+
+An important thing to consider about our implementation of the Null Count is
+that we need to support both `undefined` and `nil` as null values as they are
+the conventions for null values in Erlang, and Elixir respectively.
+
+There are 5 important characteristics to remember about the validity bitmap:
--include("arrow_buffer.hrl").
+1. A null value is represented by a 0 bit, and a non null value by a 1 bit.
+
+2. Every 8 elements's validities are batched into a byte, which are then
+ reversed as Arrow uses least-significant bit (LSB) numbering (more in
+ attached reference).
+
+3. If a "batch" consists of less than 8 elements, its validity needs to be
+ padded by 0 bits so that it can make a byte.
+
+4. Each byte is stored in a slot of a Buffer (see docs for `arrow_buffer`). This
+ buffer with the validities of each batch of 8 elements make up what is called
+ the Validity Bitmap.
+
+5. If the Null Count is 0, we can allocate the Validity Bitmap as a NULL pointer
+ (which in Erlang's case is `undefined`).
+""".
+-export([validity_bitmap/1]).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Validity Bitmap & Null Count %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%% @doc Returns the Validity Bitmap along with the Null Count, of
-%% an Array.
+-doc """
+Returns the Validity Bitmap along with the Null Count, of an Array.
+""".
-spec validity_bitmap(Value :: [arrow_type:native_type()] | list()) ->
- {Bitmap :: #buffer{}, non_neg_integer()}.
+ {Bitmap :: arrow_buffer:buffer(), non_neg_integer()}.
Review Comment:
This makes sense. Surprisingly, dialyzer isn't raising warnings. We'll add a TODO and ignore it for now. Maybe we should run eqWAlizer via ELP on the CI to catch more errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org