You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2018/09/30 22:08:48 UTC
[05/12] arrow-site git commit: Update to latest website
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/bd12ebcd/docs/ipc.html
----------------------------------------------------------------------
diff --git a/docs/ipc.html b/docs/ipc.html
index 4a71096..2a9fdb7 100644
--- a/docs/ipc.html
+++ b/docs/ipc.html
@@ -5,7 +5,7 @@
<title>Apache Arrow Homepage</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="generator" content="Jekyll v3.4.3">
+ <meta name="generator" content="Jekyll v3.8.4">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<link rel="icon" type="image/x-icon" href="/favicon.ico">
@@ -13,10 +13,8 @@
<link href="/css/main.css" rel="stylesheet">
<link href="/css/syntax.css" rel="stylesheet">
- <script src="https://code.jquery.com/jquery-3.2.1.min.js"
- integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
- crossorigin="anonymous"></script>
- <script src="/assets/javascripts/bootstrap.min.js"></script>
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
<!-- Global Site Tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
@@ -33,96 +31,98 @@
<body class="wrap">
- <div class="container">
- <nav class="navbar navbar-default">
- <div class="container-fluid">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#arrow-navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a class="navbar-brand" href="/">Apache Arrow™ </a>
- </div>
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+ <a class="navbar-brand" href="/">Apache Arrow™ </a>
+ <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="arrow-navbar">
<ul class="nav navbar-nav">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Project Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownProjectLinks" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Project Links
</a>
- <ul class="dropdown-menu">
- <li><a href="/install/">Install</a></li>
- <li><a href="/blog/">Blog</a></li>
- <li><a href="/release/">Releases</a></li>
- <li><a href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a></li>
- <li><a href="https://github.com/apache/arrow">Source Code</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownProjectLinks">
+ <a class="dropdown-item" href="/install/">Install</a>
+ <a class="dropdown-item" href="/blog/">Blog</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Community<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
</a>
- <ul class="dropdown-menu">
- <li><a href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Mailing List</a></li>
- <li><a href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a></li>
- <li><a href="/committers/">Committers</a></li>
- <li><a href="/powered_by/">Powered By</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-user/">User Mailing List</a>
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Dev Mailing List</a>
+ <a class="dropdown-item" href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a>
+ <a class="dropdown-item" href="/committers/">Committers</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Specification<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownSpecification" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Specification
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/memory_layout.html">Memory Layout</a></li>
- <li><a href="/docs/metadata.html">Metadata</a></li>
- <li><a href="/docs/ipc.html">Messaging / IPC</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownSpecification">
+ <a class="dropdown-item" href="/docs/memory_layout.html">Memory Layout</a>
+ <a class="dropdown-item" href="/docs/metadata.html">Metadata</a>
+ <a class="dropdown-item" href="/docs/ipc.html">Messaging / IPC</a>
+ </div>
</li>
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Documentation<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/python">Python</a></li>
- <li><a href="/docs/cpp">C++ API</a></li>
- <li><a href="/docs/java">Java API</a></li>
- <li><a href="/docs/c_glib">C GLib API</a></li>
- <li><a href="/docs/js">Javascript API</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/cpp">C++ API</a>
+ <a class="dropdown-item" href="/docs/java">Java API</a>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib API</a>
+ <a class="dropdown-item" href="/docs/js">Javascript API</a>
+ </div>
</li>
<!-- <li><a href="/blog">Blog</a></li> -->
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">ASF Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/">ASF Website</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Donate</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
</li>
</ul>
- <a href="http://www.apache.org/">
+ <a href="http://www.apache.org/" class="flex-row ml-md-auto">
<img style="float:right;" src="/img/asf_logo.svg" width="120px"/>
</a>
</div><!-- /.navbar-collapse -->
</div>
</nav>
+ </header>
- <!--
+ <div class="container p-lg-4">
+ <main role="main">
+ <!--
-->
@@ -161,12 +161,11 @@
<p>Schematically, we have:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><metadata_size: int32>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><metadata_size: int32>
<metadata_flatbuffer: bytes>
<padding>
<message body>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The complete serialized message must be a multiple of 8 bytes so that messages
can be relocated between streams. Otherwise the amount of padding between the
@@ -176,13 +175,12 @@ metadata and the message body could be non-deterministic.</p>
<code class="highlighter-rouge">Message</code> flatbuffer includes a version number, the particular message (as a
flatbuffer union), and the size of the message body:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>table Message {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>table Message {
version: org.apache.arrow.flatbuf.MetadataVersion;
header: MessageHeader;
bodyLength: long;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Currently, we support 4 types of messages:</p>
@@ -203,7 +201,7 @@ that follow. If any fields in the schema are dictionary-encoded, one or more
<code class="highlighter-rouge">RecordBatch</code> messages may be interleaved, but before any dictionary key is used
in a <code class="highlighter-rouge">RecordBatch</code> it should be defined in a <code class="highlighter-rouge">DictionaryBatch</code>.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><SCHEMA>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><SCHEMA>
<DICTIONARY 0>
...
<DICTIONARY k - 1>
@@ -215,8 +213,7 @@ in a <code class="highlighter-rouge">RecordBatch</code> it should be defined in
...
<RECORD BATCH n - 1>
<EOS [optional]: int32>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>When a stream reader implementation is reading a stream, after each message, it
may read the next 4 bytes to know how large the message metadata that follows
@@ -238,14 +235,13 @@ footer.</p>
<p>Schematically we have:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><magic number "ARROW1">
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><magic number "ARROW1">
<empty padding bytes [to 8 byte boundary]>
<STREAMING FORMAT>
<FOOTER>
<FOOTER SIZE: int32>
<magic number "ARROW1">
-</code></pre>
-</div>
+</code></pre></div></div>
<p>In the file format, there is no requirement that dictionary keys should be
defined in a <code class="highlighter-rouge">DictionaryBatch</code> before they are used in a <code class="highlighter-rouge">RecordBatch</code>, as long
@@ -257,7 +253,7 @@ as the keys are defined somewhere in the file.</p>
field metadata and physical memory buffers (some comments from <a href="https://github.com/apache/arrow/blob/master/format/Message.fbs">Message.fbs</a>
have been shortened / removed):</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>table RecordBatch {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>table RecordBatch {
length: long;
nodes: [FieldNode];
buffers: [Buffer];
@@ -269,10 +265,6 @@ struct FieldNode {
}
struct Buffer {
- /// The shared memory page id where this buffer is located. Currently this is
- /// not used
- page: int;
-
/// The relative offset into the shared memory page where the bytes for this
/// buffer starts
offset: long;
@@ -281,8 +273,7 @@ struct Buffer {
/// from offset (inclusive) to offset + length (non-inclusive).
length: long;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>In the context of a file, the <code class="highlighter-rouge">page</code> is not used, and the <code class="highlighter-rouge">Buffer</code> offsets use
as a frame of reference the start of the message body. So, while in a general
@@ -292,13 +283,12 @@ in the file format the offsets start from 0.</p>
<p>The location of a record batch and the size of the metadata block as well as
the body of buffers is stored in the file footer:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>struct Block {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct Block {
offset: long;
metaDataLength: int;
bodyLength: long;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The <code class="highlighter-rouge">metaDataLength</code> here includes the metadata length prefix, serialized
metadata, and any additional padding bytes, and by construction must be a
@@ -321,13 +311,12 @@ the dictionaries. The dictionary types are found in the schema, so it is
necessary to read the schema to first determine the dictionary types so that
the dictionaries can be properly interpreted.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>table DictionaryBatch {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>table DictionaryBatch {
id: long;
data: RecordBatch;
isDelta: boolean = false;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The dictionary <code class="highlighter-rouge">id</code> in the message metadata can be referenced one or more times
in the schema, so that dictionaries can even be used for multiple fields. See
@@ -341,7 +330,7 @@ stream which encodes one column, the list of strings
<code class="highlighter-rouge">["A", "B", "C", "B", "D", "C", "E", "A"]</code>, with a delta dictionary batch could
take the form:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><SCHEMA>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><SCHEMA>
<DICTIONARY 0>
(0) "A"
(1) "B"
@@ -363,8 +352,7 @@ take the form:</p>
4
0
EOS
-</code></pre>
-</div>
+</code></pre></div></div>
<h3 id="tensor-multi-dimensional-array-message-format">Tensor (Multi-dimensional Array) Message Format</h3>
@@ -378,23 +366,23 @@ indicated above, but additionally align the starting offset of the metadata as
well as the starting offset of the tensor body (if writing to a shared memory
region) to be multiples of 64 bytes:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><PADDING>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><PADDING>
<metadata size: int32>
<metadata>
<tensor body>
-</code></pre>
-</div>
+</code></pre></div></div>
-
- <hr></hr>
+ </main>
<hr/>
<footer class="footer">
<p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
<p>© 2017 Apache Software Foundation</p>
+ <script src="/assets/main-8d2a359fd27a888246eb638b36a4e8b68ac65b9f11c48b9fac601fa0c9a7d796.js" integrity="sha256-jSo1n9J6iIJG62OLNqTotorGW58RxIufrGAfoMmn15Y=" crossorigin="anonymous" type="text/javascript"></script>
</footer>
</div>
</body>
</html>
+
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/bd12ebcd/docs/memory_layout.html
----------------------------------------------------------------------
diff --git a/docs/memory_layout.html b/docs/memory_layout.html
index 6014f7c..a36b36f 100644
--- a/docs/memory_layout.html
+++ b/docs/memory_layout.html
@@ -5,7 +5,7 @@
<title>Apache Arrow Homepage</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="generator" content="Jekyll v3.4.3">
+ <meta name="generator" content="Jekyll v3.8.4">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<link rel="icon" type="image/x-icon" href="/favicon.ico">
@@ -13,10 +13,8 @@
<link href="/css/main.css" rel="stylesheet">
<link href="/css/syntax.css" rel="stylesheet">
- <script src="https://code.jquery.com/jquery-3.2.1.min.js"
- integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
- crossorigin="anonymous"></script>
- <script src="/assets/javascripts/bootstrap.min.js"></script>
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
<!-- Global Site Tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
@@ -33,96 +31,98 @@
<body class="wrap">
- <div class="container">
- <nav class="navbar navbar-default">
- <div class="container-fluid">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#arrow-navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a class="navbar-brand" href="/">Apache Arrow™ </a>
- </div>
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+ <a class="navbar-brand" href="/">Apache Arrow™ </a>
+ <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="arrow-navbar">
<ul class="nav navbar-nav">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Project Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownProjectLinks" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Project Links
</a>
- <ul class="dropdown-menu">
- <li><a href="/install/">Install</a></li>
- <li><a href="/blog/">Blog</a></li>
- <li><a href="/release/">Releases</a></li>
- <li><a href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a></li>
- <li><a href="https://github.com/apache/arrow">Source Code</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownProjectLinks">
+ <a class="dropdown-item" href="/install/">Install</a>
+ <a class="dropdown-item" href="/blog/">Blog</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Community<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
</a>
- <ul class="dropdown-menu">
- <li><a href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Mailing List</a></li>
- <li><a href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a></li>
- <li><a href="/committers/">Committers</a></li>
- <li><a href="/powered_by/">Powered By</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-user/">User Mailing List</a>
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Dev Mailing List</a>
+ <a class="dropdown-item" href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a>
+ <a class="dropdown-item" href="/committers/">Committers</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Specification<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownSpecification" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Specification
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/memory_layout.html">Memory Layout</a></li>
- <li><a href="/docs/metadata.html">Metadata</a></li>
- <li><a href="/docs/ipc.html">Messaging / IPC</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownSpecification">
+ <a class="dropdown-item" href="/docs/memory_layout.html">Memory Layout</a>
+ <a class="dropdown-item" href="/docs/metadata.html">Metadata</a>
+ <a class="dropdown-item" href="/docs/ipc.html">Messaging / IPC</a>
+ </div>
</li>
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Documentation<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/python">Python</a></li>
- <li><a href="/docs/cpp">C++ API</a></li>
- <li><a href="/docs/java">Java API</a></li>
- <li><a href="/docs/c_glib">C GLib API</a></li>
- <li><a href="/docs/js">Javascript API</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/cpp">C++ API</a>
+ <a class="dropdown-item" href="/docs/java">Java API</a>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib API</a>
+ <a class="dropdown-item" href="/docs/js">Javascript API</a>
+ </div>
</li>
<!-- <li><a href="/blog">Blog</a></li> -->
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">ASF Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/">ASF Website</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Donate</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
</li>
</ul>
- <a href="http://www.apache.org/">
+ <a href="http://www.apache.org/" class="flex-row ml-md-auto">
<img style="float:right;" src="/img/asf_logo.svg" width="120px"/>
</a>
</div><!-- /.navbar-collapse -->
</div>
</nav>
+ </header>
- <!--
+ <div class="container p-lg-4">
+ <main role="main">
+ <!--
-->
@@ -305,21 +305,19 @@ this bitmap. A 1 (set bit) for index <code class="highlighter-rouge">j</code> in
while a 0 (bit not set) indicates that it is null. Bitmaps are to be
initialized to be all unset at allocation time (this includes padding).</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>is_valid[j] -> bitmap[j / 8] & (1 << (j % 8))
-</code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>is_valid[j] -> bitmap[j / 8] & (1 << (j % 8))
+</code></pre></div></div>
<p>We use <a href="https://en.wikipedia.org/wiki/Bit_numbering">least-significant bit (LSB) numbering</a> (also known as
bit-endianness). This means that within a group of 8 bits, we read
right-to-left:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>values = [0, 1, null, 2, null, 3]
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>values = [0, 1, null, 2, null, 3]
bitmap
j mod 8 7 6 5 4 3 2 1 0
0 0 1 0 1 0 1 1
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Arrays having a 0 null count may choose to not allocate the null
bitmap. Implementations may choose to always allocate one anyway as a matter of
@@ -348,7 +346,7 @@ does not need to be adjacent in memory to the values buffer.</p>
<p>Would look like:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length: 5, Null count: 1
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length: 5, Null count: 1
* Null bitmap buffer:
|Byte 0 (validity bitmap) | Bytes 1-63 |
@@ -360,14 +358,13 @@ does not need to be adjacent in memory to the values buffer.</p>
|Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-19 | Bytes 20-63 |
|------------|-------------|-------------|-------------|-------------|-------------|
| 1 | unspecified | 2 | 4 | 8 | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<h3 id="example-layout-non-null-int32-array">Example Layout: Non-null int32 Array</h3>
<p>[1, 2, 3, 4, 8] has two possible layouts:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length: 5, Null count: 0
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length: 5, Null count: 0
* Null bitmap buffer:
| Byte 0 (validity bitmap) | Bytes 1-63 |
@@ -379,20 +376,18 @@ does not need to be adjacent in memory to the values buffer.</p>
|Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | bytes 12-15 | bytes 16-19 | Bytes 20-63 |
|------------|-------------|-------------|-------------|-------------|-------------|
| 1 | 2 | 3 | 4 | 8 | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<p>or with the bitmap elided:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length 5, Null count: 0
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length 5, Null count: 0
* Null bitmap buffer: Not required
* Value Buffer:
|Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | bytes 12-15 | bytes 16-19 | Bytes 20-63 |
|------------|-------------|-------------|-------------|-------------|-------------|
| 1 | 2 | 3 | 4 | 8 | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="list-type">List type</h2>
@@ -417,10 +412,9 @@ of the value in each slot is computed using the first difference with the next
element in the offsets array. For example, the position and length of slot j is
computed as:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>slot_position = offsets[j]
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>slot_position = offsets[j]
slot_length = offsets[j + 1] - offsets[j] // (for 0 <= j < length)
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The first value in the offsets array is 0, and the last element is the length
of the values array.</p>
@@ -435,7 +429,7 @@ logical type.</p>
<p>will have the following representation:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length: 4, Null count: 1
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length: 4, Null count: 1
* Null bitmap buffer:
| Byte 0 (validity bitmap) | Bytes 1-63 |
@@ -455,15 +449,14 @@ logical type.</p>
| Bytes 0-6 | Bytes 7-63 |
|------------|-------------|
| joemark | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<h3 id="example-layout-listlistbyte">Example Layout: <code class="highlighter-rouge">List<List<byte>></code></h3>
<p>[[[1, 2], [3, 4]], [[5, 6, 7], null, [8]], [[9, 10]]]</p>
<p>will be be represented as follows:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length 3
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length 3
* Nulls count: 0
* Null bitmap buffer: Not required
* Offsets buffer (int32)
@@ -493,8 +486,7 @@ logical type.</p>
| Bytes 0-9 | Bytes 10-63 |
|-------------------------------|-------------|
| 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="struct-type">Struct type</h2>
@@ -512,12 +504,11 @@ A struct array must still have an allocated null bitmap, if it has one or more n
<p>For example, the struct (field names shown here as strings for illustration
purposes)</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>Struct <
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Struct <
name: String (= List<char>),
age: Int32
>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>has two child arrays, one List<char> array (layout as above) and one 4-byte
primitive value array having Int32 logical type.</char></p>
@@ -525,7 +516,7 @@ primitive value array having Int32 logical type.</char></p>
<h3 id="example-layout-structlistchar-int32">Example Layout: <code class="highlighter-rouge">Struct<List<char>, Int32></code>:</h3>
<p>The layout for [{‘joe’, 1}, {null, 2}, null, {‘mark’, 4}] would be:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length: 4, Null count: 1
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length: 4, Null count: 1
* Null bitmap buffer:
|Byte 0 (validity bitmap) | Bytes 1-63 |
@@ -571,8 +562,7 @@ primitive value array having Int32 logical type.</char></p>
|------------|-------------|-------------|-------------|-------------|
| 1 | 2 | unspecified | 4 | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<p>While a struct does not have physical storage for each of its semantic slots
(i.e. each scalar C-like struct), an entire struct slot can be set to null via
@@ -620,7 +610,7 @@ Struct2, s3: Struct3, ...></code>)</p>
<code class="highlighter-rouge">Union<f: float, i: int32></code> having the values:
[{f=1.2}, null, {f=3.4}, {i=5}]</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length: 4, Null count: 1
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length: 4, Null count: 1
* Null bitmap buffer:
|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
@@ -659,8 +649,7 @@ Struct2, s3: Struct3, ...></code>)</p>
| Bytes 0-3 | Bytes 4-63 |
|-----------|-------------|
| 5 | unspecified |
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="sparse-union-type">Sparse union type</h2>
@@ -683,7 +672,7 @@ union, it has some advantages that may be desirable in certain use cases:</p>
<p>[{u0=5}, {u1=1.2}, {u2=’joe’}, {u1=3.4}, {u0=4}, {u2=’mark’}]</p>
<p>will have the following layout:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>* Length: 6, Null count: 0
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* Length: 6, Null count: 0
* Null bitmap buffer: Not required
* Types buffer:
@@ -743,8 +732,7 @@ union, it has some advantages that may be desirable in certain use cases:</p>
| Bytes 0-7 | Bytes 8-63 |
|------------|-----------------------|
| joemark | unspecified (padding) |
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Note that nested types in a sparse union must be internally consistent
(e.g. see the List in the diagram), i.e. random access at any index j
@@ -765,7 +753,7 @@ The dictionary has the same layout as the type of the field would dictate. Each
When a Schema references a Dictionary id, it must send at least one DictionaryBatch for this id.</p>
<p>As an example, you could have the following data:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>type: List<String>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>type: List<String>
[
['a', 'b'],
@@ -777,10 +765,9 @@ When a Schema references a Dictionary id, it must send at least one DictionaryBa
['c', 'd', 'e'],
['a', 'b']
]
-</code></pre>
-</div>
+</code></pre></div></div>
<p>In dictionary-encoded form, this could appear as:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>data List<String> (dictionary-encoded, dictionary id i)
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>data List<String> (dictionary-encoded, dictionary id i)
indices: [0, 0, 0, 1, 1, 1, 0]
dictionary i
@@ -791,23 +778,23 @@ type: List<String>
['a', 'b'],
['c', 'd', 'e'],
]
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="references">References</h2>
<p>Apache Drill Documentation - <a href="https://drill.apache.org/docs/value-vectors/">Value Vectors</a></p>
-
- <hr></hr>
+ </main>
<hr/>
<footer class="footer">
<p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
<p>© 2017 Apache Software Foundation</p>
+ <script src="/assets/main-8d2a359fd27a888246eb638b36a4e8b68ac65b9f11c48b9fac601fa0c9a7d796.js" integrity="sha256-jSo1n9J6iIJG62OLNqTotorGW58RxIufrGAfoMmn15Y=" crossorigin="anonymous" type="text/javascript"></script>
</footer>
</div>
</body>
</html>
+
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/bd12ebcd/docs/metadata.html
----------------------------------------------------------------------
diff --git a/docs/metadata.html b/docs/metadata.html
index 4320434..0ed948e 100644
--- a/docs/metadata.html
+++ b/docs/metadata.html
@@ -5,7 +5,7 @@
<title>Apache Arrow Homepage</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="generator" content="Jekyll v3.4.3">
+ <meta name="generator" content="Jekyll v3.8.4">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<link rel="icon" type="image/x-icon" href="/favicon.ico">
@@ -13,10 +13,8 @@
<link href="/css/main.css" rel="stylesheet">
<link href="/css/syntax.css" rel="stylesheet">
- <script src="https://code.jquery.com/jquery-3.2.1.min.js"
- integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
- crossorigin="anonymous"></script>
- <script src="/assets/javascripts/bootstrap.min.js"></script>
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
<!-- Global Site Tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
@@ -33,96 +31,98 @@
<body class="wrap">
- <div class="container">
- <nav class="navbar navbar-default">
- <div class="container-fluid">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#arrow-navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a class="navbar-brand" href="/">Apache Arrow™ </a>
- </div>
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+ <a class="navbar-brand" href="/">Apache Arrow™ </a>
+ <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="arrow-navbar">
<ul class="nav navbar-nav">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Project Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownProjectLinks" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Project Links
</a>
- <ul class="dropdown-menu">
- <li><a href="/install/">Install</a></li>
- <li><a href="/blog/">Blog</a></li>
- <li><a href="/release/">Releases</a></li>
- <li><a href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a></li>
- <li><a href="https://github.com/apache/arrow">Source Code</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownProjectLinks">
+ <a class="dropdown-item" href="/install/">Install</a>
+ <a class="dropdown-item" href="/blog/">Blog</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Community<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
</a>
- <ul class="dropdown-menu">
- <li><a href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Mailing List</a></li>
- <li><a href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a></li>
- <li><a href="/committers/">Committers</a></li>
- <li><a href="/powered_by/">Powered By</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-user/">User Mailing List</a>
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Dev Mailing List</a>
+ <a class="dropdown-item" href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a>
+ <a class="dropdown-item" href="/committers/">Committers</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Specification<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownSpecification" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Specification
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/memory_layout.html">Memory Layout</a></li>
- <li><a href="/docs/metadata.html">Metadata</a></li>
- <li><a href="/docs/ipc.html">Messaging / IPC</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownSpecification">
+ <a class="dropdown-item" href="/docs/memory_layout.html">Memory Layout</a>
+ <a class="dropdown-item" href="/docs/metadata.html">Metadata</a>
+ <a class="dropdown-item" href="/docs/ipc.html">Messaging / IPC</a>
+ </div>
</li>
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Documentation<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/python">Python</a></li>
- <li><a href="/docs/cpp">C++ API</a></li>
- <li><a href="/docs/java">Java API</a></li>
- <li><a href="/docs/c_glib">C GLib API</a></li>
- <li><a href="/docs/js">Javascript API</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/cpp">C++ API</a>
+ <a class="dropdown-item" href="/docs/java">Java API</a>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib API</a>
+ <a class="dropdown-item" href="/docs/js">Javascript API</a>
+ </div>
</li>
<!-- <li><a href="/blog">Blog</a></li> -->
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">ASF Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/">ASF Website</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Donate</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
</li>
</ul>
- <a href="http://www.apache.org/">
+ <a href="http://www.apache.org/" class="flex-row ml-md-auto">
<img style="float:right;" src="/img/asf_logo.svg" width="120px"/>
</a>
</div><!-- /.navbar-collapse -->
</div>
</nav>
+ </header>
- <!--
+ <div class="container p-lg-4">
+ <main role="main">
+ <!--
-->
@@ -179,137 +179,115 @@ of data.</p>
<p>A schema consists of a sequence of <strong>fields</strong>, which are metadata describing
the columns. The Flatbuffers IDL for a field is:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>table Field {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>table Field {
// Name is not required, in i.e. a List
name: string;
nullable: bool;
type: Type;
- // present only if the field is dictionary encoded
- // will point to a dictionary provided by a DictionaryBatch message
- dictionary: long;
+
+ // Present only if the field is dictionary encoded
+ dictionary: DictionaryEncoding;
+
// children apply only to Nested data types like Struct, List and Union
children: [Field];
- /// layout of buffers produced for this type (as derived from the Type)
- /// does not include children
- /// each recordbatch will return instances of those Buffers.
- layout: [ VectorLayout ];
+
// User-defined metadata
custom_metadata: [ KeyValue ];
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The <code class="highlighter-rouge">type</code> is the logical type of the field. Nested types, such as List,
Struct, and Union, have a sequence of child fields.</p>
<p>A JSON representation of the schema is also provided:
Field:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"name_of_the_field"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"nullable"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
- </span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">Type</span><span class="w"> </span><span class="err">*/</span><span class="p">,</span><span class="w">
- </span><span class="nt">"children"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">Field</span><span class="w"> </span><span class="err">*/</span><span class="w"> </span><span class="p">],</span><span class="w">
- </span><span class="nt">"typeLayout"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
- </span><span class="nt">"vectors"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">VectorLayout</span><span class="w"> </span><span class="err">*/</span><span class="w"> </span><span class="p">]</span><span class="w">
- </span><span class="p">}</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
-<p>VectorLayout:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"DATA|OFFSET|VALIDITY|TYPE"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"typeBitWidth"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">int</span><span class="w"> </span><span class="err">*/</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "name_of_the_field",
+ "nullable" : false,
+ "type" : /* Type */,
+ "children" : [ /* Field */ ],
+}
+</code></pre></div></div>
+
<p>Type:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"null|struct|list|union|int|floatingpoint|utf8|binary|fixedsizebinary|bool|decimal|date|time|timestamp|interval"</span><span class="w">
- </span><span class="err">//</span><span class="w"> </span><span class="err">fields</span><span class="w"> </span><span class="err">as</span><span class="w"> </span><span class="err">defined</span><span class="w"> </span><span class="err">in</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">Flatbuffer</span><span class="w"> </span><span class="err">depending</span><span class="w"> </span><span class="err">on</span><span class="w"> </span><span class="err">the</span><span class="w"> </span><span class="err">type</span><span class="w"> </span><span class="err">name</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "null|struct|list|union|int|floatingpoint|utf8|binary|fixedsizebinary|bool|decimal|date|time|timestamp|interval"
+ // fields as defined in the Flatbuffer depending on the type name
+}
+</code></pre></div></div>
+
<p>Union:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"union"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"mode"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Sparse|Dense"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"typeIds"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">integer</span><span class="w"> </span><span class="err">*/</span><span class="w"> </span><span class="p">]</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "union",
+ "mode" : "Sparse|Dense",
+ "typeIds" : [ /* integer */ ]
+}
+</code></pre></div></div>
<p>The <code class="highlighter-rouge">typeIds</code> field in the Union are the codes used to denote each type, which
may be different from the index of the child array. This is so that the union
type ids do not have to be enumerated from 0.</p>
<p>Int:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"int"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"bitWidth"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">integer</span><span class="w"> </span><span class="err">*/</span><span class="p">,</span><span class="w">
- </span><span class="nt">"isSigned"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">boolean</span><span class="w"> </span><span class="err">*/</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "int",
+ "bitWidth" : /* integer */,
+ "isSigned" : /* boolean */
+}
+</code></pre></div></div>
<p>FloatingPoint:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"floatingpoint"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"precision"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"HALF|SINGLE|DOUBLE"</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "floatingpoint",
+ "precision" : "HALF|SINGLE|DOUBLE"
+}
+</code></pre></div></div>
<p>Decimal:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"decimal"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"precision"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">integer</span><span class="w"> </span><span class="err">*/</span><span class="p">,</span><span class="w">
- </span><span class="nt">"scale"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">integer</span><span class="w"> </span><span class="err">*/</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "decimal",
+ "precision" : /* integer */,
+ "scale" : /* integer */
+}
+</code></pre></div></div>
<p>Timestamp:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"timestamp"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"unit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"SECOND|MILLISECOND|MICROSECOND|NANOSECOND"</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "timestamp",
+ "unit" : "SECOND|MILLISECOND|MICROSECOND|NANOSECOND"
+}
+</code></pre></div></div>
<p>Date:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"unit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"DAY|MILLISECOND"</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "date",
+ "unit" : "DAY|MILLISECOND"
+}
+</code></pre></div></div>
<p>Time:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"time"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"unit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"SECOND|MILLISECOND|MICROSECOND|NANOSECOND"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"bitWidth"</span><span class="p">:</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">integer:</span><span class="w"> </span><span class="mi">32</span><span class="w"> </span><span class="err">or</span><span class="w"> </span><span class="mi">64</span><span class="w"> </span><span class="err">*/</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "time",
+ "unit" : "SECOND|MILLISECOND|MICROSECOND|NANOSECOND",
+ "bitWidth": /* integer: 32 or 64 */
+}
+</code></pre></div></div>
<p>Interval:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"interval"</span><span class="p">,</span><span class="w">
- </span><span class="nt">"unit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"YEAR_MONTH|DAY_TIME"</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "name" : "interval",
+ "unit" : "YEAR_MONTH|DAY_TIME"
+}
+</code></pre></div></div>
<p>Schema:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
- </span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
- </span><span class="err">/*</span><span class="w"> </span><span class="err">Field</span><span class="w"> </span><span class="err">*/</span><span class="w">
- </span><span class="p">]</span><span class="w">
-</span><span class="p">}</span><span class="w">
-</span></code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
+ "fields" : [
+ /* Field */
+ ]
+}
+</code></pre></div></div>
<h2 id="record-data-headers">Record data headers</h2>
@@ -321,13 +299,12 @@ not required to be the same length as the top-level arrays.</p>
describing a particular record batch is called a “data header”. Here is the
Flatbuffers IDL for a record batch data header</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>table RecordBatch {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>table RecordBatch {
length: long;
nodes: [FieldNode];
buffers: [Buffer];
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The <code class="highlighter-rouge">RecordBatch</code> metadata provides for record batches with length exceeding
2^31 - 1, but Arrow implementations are not required to implement support
@@ -350,12 +327,11 @@ virtual address space. This may include:</p>
<p>The key form of the Buffer type is:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>struct Buffer {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct Buffer {
offset: long;
length: long;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>In the context of a record batch, each field has some number of buffers
associated with it, which are derived from their physical memory layout.</p>
@@ -369,7 +345,7 @@ the logical types section.</p>
<p>The <code class="highlighter-rouge">FieldNode</code> values contain metadata about each level in a nested type
hierarchy.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>struct FieldNode {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>struct FieldNode {
/// The number of value slots in the Arrow array at this level of a nested
/// tree
length: long;
@@ -377,8 +353,7 @@ hierarchy.</p>
/// The number of observed nulls.
null_count: lohng;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The <code class="highlighter-rouge">FieldNode</code> metadata provides for fields with length exceeding 2^31 - 1,
but Arrow implementations are not required to implement support for large
@@ -393,26 +368,24 @@ its children) are appended to the <code class="highlighter-rouge">buffers</code>
<p>For example, let’s consider the schema</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>col1: Struct<a: Int32, b: List<Int64>, c: Float64>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>col1: Struct<a: Int32, b: List<Int64>, c: Float64>
col2: Utf8
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The flattened version of this is:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>FieldNode 0: Struct name='col1'
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FieldNode 0: Struct name='col1'
FieldNode 1: Int32 name=a'
FieldNode 2: List name='b'
FieldNode 3: Int64 name='item' # arbitrary
FieldNode 4: Float64 name='c'
FieldNode 5: Utf8 name='col2'
-</code></pre>
-</div>
+</code></pre></div></div>
<p>For the buffers produced, we would have the following (as described in more
detail for each type below):</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>buffer 0: field 0 validity bitmap
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>buffer 0: field 0 validity bitmap
buffer 1: field 1 validity bitmap
buffer 2: field 1 values <int32_t*>
@@ -429,8 +402,7 @@ buffer 8: field 4 values <double*>
buffer 9: field 5 validity bitmap
buffer 10: field 5 offsets <int32_t*>
buffer 11: field 5 data <uint8_t*>
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="logical-types">Logical types</h2>
@@ -457,12 +429,11 @@ standard C integer types, both signed and unsigned:</p>
<p>The IDL looks like:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>table Int {
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>table Int {
bitWidth: int;
is_signed: bool;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The integer endianness is currently set globally at the schema level. If a
schema is set to be little-endian, then all integer types occurring within must
@@ -482,13 +453,12 @@ record batch.</p>
<p>The IDL looks like:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>enum Precision:int {HALF, SINGLE, DOUBLE}
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enum Precision:int {HALF, SINGLE, DOUBLE}
table FloatingPoint {
precision: Precision;
}
-</code></pre>
-</div>
+</code></pre></div></div>
<h3 id="boolean">Boolean</h3>
@@ -565,15 +535,16 @@ indicated unit. For second and millisecond: 32-bit, for the others 64-bit.</p>
<h2 id="dictionary-encoding">Dictionary encoding</h2>
-
- <hr></hr>
+ </main>
<hr/>
<footer class="footer">
<p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
<p>© 2017 Apache Software Foundation</p>
+ <script src="/assets/main-8d2a359fd27a888246eb638b36a4e8b68ac65b9f11c48b9fac601fa0c9a7d796.js" integrity="sha256-jSo1n9J6iIJG62OLNqTotorGW58RxIufrGAfoMmn15Y=" crossorigin="anonymous" type="text/javascript"></script>
</footer>
</div>
</body>
</html>
+