You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2018/09/30 22:08:47 UTC
[04/12] arrow-site git commit: Update to latest website
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/bd12ebcd/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index c208bc1..a6b030c 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.4.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2018-08-23T11:58:53-04:00</updated><id>/</id><entry><title type="html">Apache Arrow 0.10.0 Release</title><link href="/blog/2018/08/07/0.10.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.10.0 Release" /><published>2018-08-07T00:00:00-04:00</published><updated>2018-08-07T00:00:00-04:00</updated><id>/blog/2018/08/07/0.10.0-release</id><content type="html" xml:base="/blog/2018/08/07/0.10.0-release/"><!--
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.8.4">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2018-09-30T18:07:53-04:00</updated><id>/feed.xml</id><entry><title type="html">Apache Arrow 0.10.0 Release</title><link href="/blog/2018/08/07/0.10.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.10.0 Release" /><published>2018-08-07T00:00:00-04:00</published><updated>2018-08-07T00:00:00-04:00</updated><id>/blog/2018/08/07/0.10.0-release</id><content type="html" xml:base="/blog/2018/08/07/0.10.0-release/"><!--
-->
@@ -81,7 +81,7 @@ to expand our allocations as the data is loaded.</p>
<p>To incrementally build a vector using memory expansion of a factor of 2, we
would use the following C-code with the standard POSIX APIs:</p>
-<div class="language-c highlighter-rouge"><pre class="highlight"><code><span class="kt">size_t</span> <span class="n">size</span> <span class="o">=</span> <span class="mi">128</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">;</span>
+<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">size_t</span> <span class="n">size</span> <span class="o">=</span> <span class="mi">128</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">;</span>
<span class="kt">void</span><span class="o">*</span> <span class="n">ptr</span> <span class="o">=</span> <span class="n">aligned_alloc</span><span class="p">(</span><span class="mi">64</span><span class="p">,</span> <span class="n">size</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">size_t</span> <span class="n">new_size</span> <span class="o">=</span> <span class="n">size</span> <span class="o">*</span> <span class="mi">2</span><span class="p">;</span>
@@ -92,23 +92,21 @@ would use the following C-code with the standard POSIX APIs:</p>
<span class="n">size</span> <span class="o">=</span> <span class="n">new_size</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">free</span><span class="p">(</span><span class="n">ptr</span><span class="p">);</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>With jemalloc’s special APIs, we are able to omit the explicit call to <code class="highlighter-rouge">memcpy</code>.
In the case where a memory expansion cannot be done in-place, it is still called
by the allocator but not needed on all occasions. This simplifies our user code
to:</p>
-<div class="language-c highlighter-rouge"><pre class="highlight"><code><span class="kt">size_t</span> <span class="n">size</span> <span class="o">=</span> <span class="mi">128</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">;</span>
+<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">size_t</span> <span class="n">size</span> <span class="o">=</span> <span class="mi">128</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">;</span>
<span class="kt">void</span><span class="o">*</span> <span class="n">ptr</span> <span class="o">=</span> <span class="n">mallocx</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">MALLOCX_ALIGN</span><span class="p">(</span><span class="mi">64</span><span class="p">));</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">size</span> <span class="o">*=</span> <span class="mi">2</span><span class="p">;</span>
<span class="n">ptr</span> <span class="o">=</span> <span class="n">rallocx</span><span class="p">(</span><span class="n">ptr</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">MALLOCX_ALIGN</span><span class="p">(</span><span class="mi">64</span><span class="p">));</span>
<span class="p">}</span>
<span class="n">dallocx</span><span class="p">(</span><span class="n">ptr</span><span class="p">,</span> <span class="n">MALLOCX_ALIGN</span><span class="p">(</span><span class="mi">64</span><span class="p">));</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>To see the real world benefits of using jemalloc, we look at the benchmarks in
Arrow C++. There we have modeled a typical use case of incrementally building up
@@ -120,25 +118,23 @@ of the <code class="highlighter-rouge">builder-benchmark</cod
<p>Runtimes without <code class="highlighter-rouge">jemalloc</code>:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>BM_BuildPrimitiveArrayNoNulls/repeats:3 636726 us 804.114MB/s
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BM_BuildPrimitiveArrayNoNulls/repeats:3 636726 us 804.114MB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3 621345 us 824.019MB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3 625008 us 819.19MB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3_mean 627693 us 815.774MB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3_median 625008 us 819.19MB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3_stddev 8034 us 10.3829MB/s
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Runtimes with <code class="highlighter-rouge">jemalloc</code>:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>BM_BuildPrimitiveArrayNoNulls/repeats:3 630881 us 811.563MB/s
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BM_BuildPrimitiveArrayNoNulls/repeats:3 630881 us 811.563MB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3 352891 us 1.41687GB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3 351039 us 1.42434GB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3_mean 444937 us 1.21125GB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3_median 352891 us 1.41687GB/s
BM_BuildPrimitiveArrayNoNulls/repeats:3_stddev 161035 us 371.335MB/s
-</code></pre>
-</div>
+</code></pre></div></div>
<p>The benchmark was run three times for each configuration to see the performance
differences. The first run in each configuration yielded the same performance but
@@ -148,53 +144,7 @@ expanded in place without moving the data around. This was possible as there
were memory pages assigned to the process that were unused but not reclaimed by
the operating system. Without <code class="highlighter-rouge">jemalloc</code>, we cannot make use of them simply by
the fact that the default allocator has no API that provides aligned
-reallocation.</p></content><author><name>uwe</name></author></entry><entry><title type="html">A Native Go Library for Apache Arrow</title><link href="/blog/2018/03/22/go-code-donation/" rel="alternate" type="text/html" title="A Native Go Library for Apache Arrow" /><published>2018-03-22T00:00:00-04:00</published><updated>2018-03-22T00:00:00-04:00</updated><id>/blog/2018/03/22/go-code-donation</id><content type="html" xml:base="/blog/2018/03/22/go-code-donation/"><!--
-
--->
-
-<p>Since launching in early 2016, Apache Arrow has been growing fast. We have made
-nine major releases through the efforts of over 120 distinct contributors. The
-project’s scope has also expanded. We began by focusing on the development of
-the standardized in-memory columnar data format, which now serves as a pillar
-of the project. Since then, we have been growing into a more general
-cross-language platform for in-memory data analysis through new additions to
-the project like the <a href="http://arrow.apache.org/blog/2017/08/16/0.6.0-release/">Plasma shared memory object store</a>. A primary goal of
-the project is to enable data system developers to process and move data fast.</p>
-
-<p>So far, we officially have developed native Arrow implementations in C++, Java,
-and JavaScript. We have created binding layers for the C++ libraries in C
-(using the GLib libraries) and Python. We have also seen efforts to develop
-interfaces to the Arrow C++ libraries in Go, Lua, Ruby, and Rust. While binding
-layers serve many purposes, there can be benefits to native implementations,
-and so we’ve been keen to see future work on native implementations in growing
-systems languages like Go and Rust.</p>
-
-<p>This past October, engineers <a href="https://github.com/stuartcarnie">Stuart Carnie</a>, <a href="https://github.com/nathanielc">Nathaniel Cook</a>, and
-<a href="https://github.com/goller">Chris Goller</a>, employees of <a href="https://influxdata.com">InfluxData</a>, began developing a native [Go
-language implementation of the <a href="https://github.com/influxdata/arrow">Apache Arrow</a> in-memory columnar format for
-use in Go-based database systems like InfluxDB. We are excited to announce that
-InfluxData has <a href="https://www.businesswire.com/news/home/20180322005393/en/InfluxData-Announces-Language-Implementation-Contribution-Apache-Arrow">donated this native Go implementation to the Apache Arrow
-project</a>, where it will continue to be developed. This work features
-low-level integration with the Go runtime and native support for SIMD
-instruction sets. We are looking forward to working more closely with the Go
-community on solving in-memory analytics and data interoperability problems.</p>
-
-<div align="center">
-<img src="/img/native_go_implementation.png" alt="Apache Arrow implementations and bindings" width="60%" class="img-responsive" />
-</div>
-
-<p>One of the mantras in <a href="https://www.apache.org">The Apache Software Foundation</a> is “Community over
-Code”. By building an open and collaborative development community across many
-programming language ecosystems, we will be able to development better and
-longer-lived solutions to the systems problems faced by data developers.</p>
-
-<p>We are excited for what the future holds for the Apache Arrow project. Adding
-first-class support for a popular systems programming language like Go is an
-important step along the way. We welcome others from the Go community to get
-involved in the project. We also welcome others who wish to explore building
-Arrow support for other programming languages not yet represented. Learn more
-at <a href="https://arrow.apache.org">https://arrow.apache.org</a> and join the mailing list
-<a href="https://lists.apache.org/list.html?dev@arrow.apache.org">dev@arrow.apache.org</a>.</p></content><author><name>pmc</name></author></entry><entry><title type="html">Apache Arrow 0.9.0 Release</title><link href="/blog/2018/03/22/0.9.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.9.0 Release" /><published>2018-03-22T00:00:00-04:00</published><updated>2018-03-22T00:00:00-04:00</updated><id>/blog/2018/03/22/0.9.0-release</id><content type="html" xml:base="/blog/2018/03/22/0.9.0-release/"><!--
+reallocation.</p></content><author><name>uwe</name></author></entry><entry><title type="html">Apache Arrow 0.9.0 Release</title><link href="/blog/2018/03/22/0.9.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.9.0 Release" /><published>2018-03-22T00:00:00-04:00</published><updated>2018-03-22T00:00:00-04:00</updated><id>/blog/2018/03/22/0.9.0-release</id><content type="html" xml:base="/blog/2018/03/22/0.9.0-release/"><!--
-->
@@ -263,7 +213,53 @@ which are being <a href="https://www.npmjs.com/package/apache-arrow"
<p>In the coming months, we will be working to move Apache Arrow closer to a 1.0.0
release. We will also be discussing plans to develop native Arrow-based
-computational libraries within the project.</p></content><author><name>wesm</name></author></entry><entry><title type="html">Apache Arrow 0.8.0 Release</title><link href="/blog/2017/12/18/0.8.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.8.0 Release" /><published>2017-12-18T23:01:00-05:00</published><updated>2017-12-18T23:01:00-05:00</updated><id>/blog/2017/12/18/0.8.0-release</id><content type="html" xml:base="/blog/2017/12/18/0.8.0-release/"><!--
+computational libraries within the project.</p></content><author><name>wesm</name></author></entry><entry><title type="html">A Native Go Library for Apache Arrow</title><link href="/blog/2018/03/22/go-code-donation/" rel="alternate" type="text/html" title="A Native Go Library for Apache Arrow" /><published>2018-03-22T00:00:00-04:00</published><updated>2018-03-22T00:00:00-04:00</updated><id>/blog/2018/03/22/go-code-donation</id><content type="html" xml:base="/blog/2018/03/22/go-code-donation/"><!--
+
+-->
+
+<p>Since launching in early 2016, Apache Arrow has been growing fast. We have made
+nine major releases through the efforts of over 120 distinct contributors. The
+project’s scope has also expanded. We began by focusing on the development of
+the standardized in-memory columnar data format, which now serves as a pillar
+of the project. Since then, we have been growing into a more general
+cross-language platform for in-memory data analysis through new additions to
+the project like the <a href="http://arrow.apache.org/blog/2017/08/16/0.6.0-release/">Plasma shared memory object store</a>. A primary goal of
+the project is to enable data system developers to process and move data fast.</p>
+
+<p>So far, we officially have developed native Arrow implementations in C++, Java,
+and JavaScript. We have created binding layers for the C++ libraries in C
+(using the GLib libraries) and Python. We have also seen efforts to develop
+interfaces to the Arrow C++ libraries in Go, Lua, Ruby, and Rust. While binding
+layers serve many purposes, there can be benefits to native implementations,
+and so we’ve been keen to see future work on native implementations in growing
+systems languages like Go and Rust.</p>
+
+<p>This past October, engineers <a href="https://github.com/stuartcarnie">Stuart Carnie</a>, <a href="https://github.com/nathanielc">Nathaniel Cook</a>, and
+<a href="https://github.com/goller">Chris Goller</a>, employees of <a href="https://influxdata.com">InfluxData</a>, began developing a native [Go
+language implementation of the <a href="https://github.com/influxdata/arrow">Apache Arrow</a> in-memory columnar format for
+use in Go-based database systems like InfluxDB. We are excited to announce that
+InfluxData has <a href="https://www.businesswire.com/news/home/20180322005393/en/InfluxData-Announces-Language-Implementation-Contribution-Apache-Arrow">donated this native Go implementation to the Apache Arrow
+project</a>, where it will continue to be developed. This work features
+low-level integration with the Go runtime and native support for SIMD
+instruction sets. We are looking forward to working more closely with the Go
+community on solving in-memory analytics and data interoperability problems.</p>
+
+<div align="center">
+<img src="/img/native_go_implementation.png" alt="Apache Arrow implementations and bindings" width="60%" class="img-responsive" />
+</div>
+
+<p>One of the mantras in <a href="https://www.apache.org">The Apache Software Foundation</a> is “Community over
+Code”. By building an open and collaborative development community across many
+programming language ecosystems, we will be able to development better and
+longer-lived solutions to the systems problems faced by data developers.</p>
+
+<p>We are excited for what the future holds for the Apache Arrow project. Adding
+first-class support for a popular systems programming language like Go is an
+important step along the way. We welcome others from the Go community to get
+involved in the project. We also welcome others who wish to explore building
+Arrow support for other programming languages not yet represented. Learn more
+at <a href="https://arrow.apache.org">https://arrow.apache.org</a> and join the mailing list
+<a href="https://lists.apache.org/list.html?dev@arrow.apache.org">dev@arrow.apache.org</a>.</p></content><author><name>pmc</name></author></entry><entry><title type="html">Apache Arrow 0.8.0 Release</title><link href="/blog/2017/12/18/0.8.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.8.0 Release" /><published>2017-12-18T23:01:00-05:00</published><updated>2017-12-18T23:01:00-05:00</updated><id>/blog/2017/12/18/0.8.0-release</id><content type="html" xml:base="/blog/2017/12/18/0.8.0-release/"><!--
-->
@@ -416,7 +412,7 @@ working to improve and expand the libraries in support of downstream use cases.&
<p>We continue to look for more JavaScript, Julia, R, Rust, and other programming
language developers to join the project and expand the available
-implementations and bindings to more languages.</p></content><author><name>wesm</name></author></entry><entry><title type="html">Improvements to Java Vector API in Apache Arrow 0.8.0</title><link href="/blog/2017/12/19/java-vector-improvements/" rel="alternate" type="text/html" title="Improvements to Java Vector API in Apache Arrow 0.8.0" /><published>2017-12-18T19:00:00-05:00</published><updated>2017-12-18T19:00:00-05:00</updated><id>/blog/2017/12/19/java-vector-improvements</id><content type="html" xml:base="/blog/2017/12/19/java-vector-improvements/"><!--
+implementations and bindings to more languages.</p></content><author><name>wesm</name></author></entry><entry><title type="html">Improvements to Java Vector API in Apache Arrow 0.8.0</title><link href="/blog/2017/12/18/java-vector-improvements/" rel="alternate" type="text/html" title="Improvements to Java Vector API in Apache Arrow 0.8.0" /><published>2017-12-18T19:00:00-05:00</published><updated>2017-12-18T19:00:00-05:00</updated><id>/blog/2017/12/18/java-vector-improvements</id><content type="html" xml:base="/blog/2017/12/18/java-vector-improvements/"><!--
-->
@@ -618,19 +614,17 @@ Python classes and deeply nested objects.</p>
<p>The serialization library can be used directly through pyarrow as follows. More
documentation is available <a href="https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization">here</a>.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="n">x</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="s">'hello'</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">5.0</span><span class="p"
;>,</span> <span class="mf">6.0</span><span class="p">])]</span>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">x</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="s">'hello'</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">5.0&
lt;/span><span class="p">,</span> <span class="mf">6.0</span><span class="p">])]</span>
<span class="n">serialized_x</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">serialize</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">to_buffer</span><span class="p">()</span>
<span class="n">deserialized_x</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">deserialize</span><span class="p">(</span><span class="n">serialized_x</span><span class="p">)</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>It can be used directly through the Ray API as follows.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="n">x</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="s">'hello'</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">5.0</span><span class="p"
;>,</span> <span class="mf">6.0</span><span class="p">])]</span>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">x</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="s">'hello'</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">5.0&
lt;/span><span class="p">,</span> <span class="mf">6.0</span><span class="p">])]</span>
<span class="n">x_id</span> <span class="o">=</span> <span class="n">ray</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">deserialized_x</span> <span class="o">=</span> <span class="n">ray</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">x_id</span><span class="p">)</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="data-representation">Data Representation</h2>
@@ -649,21 +643,19 @@ serialized object, and the UnionArray contains references to these tensors.</
<p>To give a concrete example, consider the following object.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="s">'hello'</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">5.0</span><span class="p">,</span> <span class="mf">6.0</span><span class="p&qu
ot;>])]</span>
-</code></pre>
-</div>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="s">'hello'</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">5.0</span><span class="p">,</span> <span class="mf">6.
0</span><span class="p">])]</span>
+</code></pre></div></div>
<p>It would be represented in Arrow with the following structure.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>UnionArray(type_ids=[tuple, string, int, int, ndarray],
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>UnionArray(type_ids=[tuple, string, int, int, ndarray],
tuples=ListArray(offsets=[0, 2],
UnionArray(type_ids=[int, int],
ints=[1, 2])),
strings=['hello'],
ints=[3, 4],
ndarrays=[&lt;offset of numpy array&gt;])
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Arrow uses Flatbuffers to encode serialized schemas. <strong>Using only the schema, we
can compute the offsets of each value in the data blob without scanning through
@@ -706,10 +698,10 @@ Benchmarking <code class="highlighter-rouge">ray.put</code>
<code class="highlighter-rouge">pyarrow.deserialize</code> gives similar figures. The plots were generated at this
<a href="https://github.com/apache/arrow/tree/894f7400977693b4e0e8f4b9845fd89481f6bf29">commit</a>.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pickle</span>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pickle</span>
<span class="kn">import</span> <span class="nn">pyarrow</span>
-<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="kn">as</span> <span class="nn">plt</span>
-<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span>
+<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">import</span> <span class="nn">timeit</span>
@@ -773,8 +765,7 @@ Benchmarking <code class="highlighter-rouge">ray.put</code>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">test_objects</span><span class="p">)):</span>
<span class="n">plot</span><span class="p">(</span><span class="o">*</span><span class="n">benchmark_object</span><span class="p">(</span><span class="n">test_objects</span><span class="p">[</span><span class="n">i</span><span class="p">]),</span> <span class="n">titles</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">i</span><span class="p">)</span>
-</code></pre>
-</div></content><author><name>Philipp Moritz, Robert Nishihara</name></author><summary type="html">This post describes how serialization works in Ray.</summary></entry><entry><title type="html">Apache Arrow 0.7.0 Release</title><link href="/blog/2017/09/19/0.7.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.7.0 Release" /><published>2017-09-19T00:00:00-04:00</published><updated>2017-09-19T00:00:00-04:00</updated><id>/blog/2017/09/19/0.7.0-release</id><content type="html" xml:base="/blog/2017/09/19/0.7.0-release/"><!--
+</code></pre></div></div></content><author><name>Philipp Moritz, Robert Nishihara</name></author><summary type="html">This post describes how serialization works in Ray.</summary></entry><entry><title type="html">Apache Arrow 0.7.0 Release</title><link href="/blog/2017/09/19/0.7.0-release/" rel="alternate" type="text/html" title="Apache Arrow 0.7.0 Release" /><published>2017-09-19T00:00:00-04:00</published><updated>2017-09-19T00:00:00-04:00</updated><id>/blog/2017/09/19/0.7.0-release</id><content type="html" xml:base="/blog/2017/09/19/0.7.0-release/"><!--
-->
@@ -820,7 +811,7 @@ library, we implemented a variety of type conversion functions. These functions
are essential in ETL tasks when conforming one table schema to another. These
are similar to the <code class="highlighter-rouge">astype</code> function in NumPy.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="n">In</span> <span class="p">[</span><span class="mi">17</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">pyarrow</span> <span class="kn">as</span> <span class="nn">pa</span>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">In</span> <span class="p">[</span><span class="mi">17</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="n">pa</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">18</span><span class="p">]:</span> <span class="n">arr</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="bp">True</span><span class="p">,</span> <span class="bp">False</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="bp">True</span><span class="p">])</span>
@@ -843,8 +834,7 @@ are similar to the <code class="highlighter-rouge">astype</co
<span class="n">NA</span><span class="p">,</span>
<span class="mi">1</span>
<span class="p">]</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>Over time these will expand to support as many input-and-output type
combinations with optimized conversions.</p>
@@ -860,7 +850,7 @@ columnar memory.</p>
<p>For example, we could write a record batch from CPU memory to GPU device memory
like so (some error checking omitted):</p>
-<div class="language-c++ highlighter-rouge"><pre class="highlight"><code><span class="cp">#include &lt;arrow/api.h&gt;
+<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;arrow/api.h&gt;
#include &lt;arrow/gpu/cuda_api.h&gt;
</span>
<span class="k">using</span> <span class="k">namespace</span> <span class="n">arrow</span><span class="p">;</span>
@@ -875,20 +865,18 @@ like so (some error checking omitted):</p>
<span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o">&lt;</span><span class="n">gpu</span><span class="o">::</span><span class="n">CudaBuffer</span><span class="o">&gt;</span> <span class="n">device_serialized</span><span class="p">;</span>
<span class="n">gpu</span><span class="o">::</span><span class="n">SerializeRecordBatch</span><span class="p">(</span><span class="o">*</span><span class="n">batch</span><span class="p">,</span> <span class="n">context_</span><span class="p">.</span><span class="n">get</span><span class="p">(),</span> <span class="o">&amp;</span><span class="n">device_serialized</span><span class="p">));</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>We can then “read” the GPU record batch, but the returned <code class="highlighter-rouge">arrow::RecordBatch</code>
internally will contain GPU device pointers that you can use for CUDA kernel
calls:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>std::shared_ptr&lt;RecordBatch&gt; device_batch;
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>std::shared_ptr&lt;RecordBatch&gt; device_batch;
gpu::ReadRecordBatch(batch-&gt;schema(), device_serialized,
default_memory_pool(), &amp;device_batch));
// Now run some CUDA kernels on device_batch
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="decimal-integration-tests">Decimal Integration Tests</h2>
@@ -973,13 +961,12 @@ be done in a future release.</p>
<p>As an example, the Python data:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>data = [{'a': 1, 'bb': 2, 'cc': 3}, {'dddd': 4}]
-</code></pre>
-</div>
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>data = [{'a': 1, 'bb': 2, 'cc': 3}, {'dddd': 4}]
+</code></pre></div></div>
<p>Could be represented in an Arrow <code class="highlighter-rouge">Map&lt;String, Int32&gt;</code> as:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>Map&lt;String, Int32&gt; = List&lt;Struct&lt;keys: String, values: Int32&gt;&gt;
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Map&lt;String, Int32&gt; = List&lt;Struct&lt;keys: String, values: Int32&gt;&gt;
is_valid: [true, true]
offsets: [0, 3, 4]
values: Struct&lt;keys: String, values: Int32&gt;
@@ -991,8 +978,7 @@ be done in a future release.</p>
- values: Int32
is_valid: [true, true, true, true]
data: [1, 2, 3, 4]
-</code></pre>
-</div>
+</code></pre></div></div>
<h2 id="python-changes">Python Changes</h2>
<p>Some highlights of Python development outside of bug fixes and general API
@@ -1070,7 +1056,7 @@ the client can write to the buffer and construct the object within the allocated
buffer. When the client is done, the client <em>seals</em> the buffer making the object
immutable and making it available to other Plasma clients.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="c"># Create an object.</span>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create an object.</span>
<span class="n">object_id</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="mi">20</span> <span class="o">*</span> <span class="n">b</span><span class="s">'a'</span><span class="p">)</span>
<span class="n">object_size</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="nb">buffer</span> <span class="o">=</span> <span class="n">memoryview</span><span class="p">(</span><span class="n">client</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">object_id</span><span class="p">,</span> <span class="n">object_size</span><span class="p">))</span>
@@ -1081,18 +1067,16 @@ immutable and making it available to other Plasma clients.</p>
<span class="c"># Seal the object making it immutable and available to other clients.</span>
<span class="n">client</span><span class="o">.</span><span class="n">seal</span><span class="p">(</span><span class="n">object_id</span><span class="p">)</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p><strong>Getting an object:</strong> After an object has been sealed, any client who knows the
object ID can get the object.</p>
-<div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="c"># Get the object from the store. This blocks until the object has been sealed.</span>
+<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Get the object from the store. This blocks until the object has been sealed.</span>
<span class="n">object_id</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">plasma</span><span class="o">.</span><span class="n">ObjectID</span><span class="p">(</span><span class="mi">20</span> <span class="o">*</span> <span class="n">b</span><span class="s">'a'</span><span class="p">)</span>
<span class="p">[</span><span class="n">buff</span><span class="p">]</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">([</span><span class="n">object_id</span><span class="p">])</span>
<span class="nb">buffer</span> <span class="o">=</span> <span class="n">memoryview</span><span class="p">(</span><span class="n">buff</span><span class="p">)</span>
-</code></pre>
-</div>
+</code></pre></div></div>
<p>If the object has not been sealed yet, then the call to <code class="highlighter-rouge">client.get</code> will block
until the object has been sealed.</p>
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/bd12ebcd/index.html
----------------------------------------------------------------------
diff --git a/index.html b/index.html
index a1d8f87..29cf9fa 100644
--- a/index.html
+++ b/index.html
@@ -5,7 +5,7 @@
<title>Apache Arrow Homepage</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <meta name="generator" content="Jekyll v3.4.3">
+ <meta name="generator" content="Jekyll v3.8.4">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<link rel="icon" type="image/x-icon" href="/favicon.ico">
@@ -13,10 +13,8 @@
<link href="/css/main.css" rel="stylesheet">
<link href="/css/syntax.css" rel="stylesheet">
- <script src="https://code.jquery.com/jquery-3.2.1.min.js"
- integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
- crossorigin="anonymous"></script>
- <script src="/assets/javascripts/bootstrap.min.js"></script>
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
<!-- Global Site Tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
@@ -33,181 +31,174 @@
<body class="wrap">
- <div class="container">
- <nav class="navbar navbar-default">
- <div class="container-fluid">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#arrow-navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
- <a class="navbar-brand" href="/">Apache Arrow™ </a>
- </div>
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+ <a class="navbar-brand" href="/">Apache Arrow™ </a>
+ <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="arrow-navbar">
<ul class="nav navbar-nav">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Project Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownProjectLinks" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Project Links
</a>
- <ul class="dropdown-menu">
- <li><a href="/install/">Install</a></li>
- <li><a href="/blog/">Blog</a></li>
- <li><a href="/release/">Releases</a></li>
- <li><a href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a></li>
- <li><a href="https://github.com/apache/arrow">Source Code</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownProjectLinks">
+ <a class="dropdown-item" href="/install/">Install</a>
+ <a class="dropdown-item" href="/blog/">Blog</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Community<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
</a>
- <ul class="dropdown-menu">
- <li><a href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Mailing List</a></li>
- <li><a href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a></li>
- <li><a href="/committers/">Committers</a></li>
- <li><a href="/powered_by/">Powered By</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-user/">User Mailing List</a>
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Dev Mailing List</a>
+ <a class="dropdown-item" href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a>
+ <a class="dropdown-item" href="/committers/">Committers</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ </div>
</li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Specification<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownSpecification" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Specification
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/memory_layout.html">Memory Layout</a></li>
- <li><a href="/docs/metadata.html">Metadata</a></li>
- <li><a href="/docs/ipc.html">Messaging / IPC</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownSpecification">
+ <a class="dropdown-item" href="/docs/memory_layout.html">Memory Layout</a>
+ <a class="dropdown-item" href="/docs/metadata.html">Metadata</a>
+ <a class="dropdown-item" href="/docs/ipc.html">Messaging / IPC</a>
+ </div>
</li>
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">Documentation<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
</a>
- <ul class="dropdown-menu">
- <li><a href="/docs/python">Python</a></li>
- <li><a href="/docs/cpp">C++ API</a></li>
- <li><a href="/docs/java">Java API</a></li>
- <li><a href="/docs/c_glib">C GLib API</a></li>
- <li><a href="/docs/js">Javascript API</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/cpp">C++ API</a>
+ <a class="dropdown-item" href="/docs/java">Java API</a>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib API</a>
+ <a class="dropdown-item" href="/docs/js">Javascript API</a>
+ </div>
</li>
<!-- <li><a href="/blog">Blog</a></li> -->
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown"
- role="button" aria-haspopup="true"
- aria-expanded="false">ASF Links<span class="caret"></span>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
</a>
- <ul class="dropdown-menu">
- <li><a href="http://www.apache.org/">ASF Website</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Donate</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- </ul>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
</li>
</ul>
- <a href="http://www.apache.org/">
+ <a href="http://www.apache.org/" class="flex-row ml-md-auto">
<img style="float:right;" src="/img/asf_logo.svg" width="120px"/>
</a>
</div><!-- /.navbar-collapse -->
</div>
</nav>
-
- <div class="container">
- <div class="jumbotron">
- <h1>Apache Arrow</h1>
- <p class="lead">A cross-language development platform for in-memory data</p>
- <p>
- <a class="btn btn-lg btn-success" style="white-space: normal;" href="mailto:dev-subscribe@arrow.apache.org" role="button">Join Mailing List</a>
- <a class="btn btn-lg btn-primary" style="white-space: normal;" href="/install/" role="button">Install (0.10.0 Release - 6 August 2018)</a>
- </p>
- </div>
- <div class="row">
- <div class="col-xs-12">
- <h4>
- Interested in contributing? Join the <a href="http://mail-archives.apache.org/mod_mbox/arrow-dev/"><strong>mailing list</strong></a> or check out the <a href="https://cwiki.apache.org/confluence/display/ARROW"><strong>developer wiki</strong></a>.
- </h4>
- </div>
- <div class="col-xs-12">
- <h4>
- <a href="/blog/"><strong>See Latest News</strong></a>
- </h4>
- </div>
- </div>
- <div class="row">
- <div class="col-xs-12">
- <p>Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, Java, JavaScript, Python, and Ruby.</p>
- </div>
- </div>
- <div class="row">
- <div class="col-lg-4">
- <h2>Fast</h2>
- <p>Apache Arrow™ enables execution engines to take advantage of the latest SIMD (Single input multiple data) operations included in modern processors, for native vectorized optimization of analytical data processing. Columnar layout is optimized for data locality for better performance on modern hardware like CPUs and GPUs.</p>
- <p>The Arrow memory format supports <strong>zero-copy reads</strong> for lightning-fast data access without serialization overhead.</p>
- </div>
- <div class="col-lg-4">
- <h2>Flexible</h2>
- <p>Arrow acts as a new high-performance interface between various systems. It is also focused on supporting a wide variety of industry-standard programming languages. Java, C, C++, Python, Ruby, and JavaScript implementations are in progress and more languages are welcome.
- </p>
- </div>
- <div class="col-lg-4">
- <h2>Standard</h2>
- <p>Apache Arrow is backed by key developers of 13 major open source projects, including Calcite, Cassandra, Drill, Hadoop, HBase, Ibis, Impala, Kudu, Pandas, Parquet, Phoenix, Spark, and Storm making it the de-facto standard for columnar in-memory analytics.</p>
- <p>Learn more about projects that are <a href="/powered_by/">Powered By Apache Arrow</a></p>
- </div>
- </div>
- <!-- close "row" div -->
-
- <div class="row">
- <div class="col-xs-12">
- <h2>Performance Advantage of Columnar In-Memory</h2>
- </div>
- <div class="col-lg-offset-2 col-lg-8 col-xs-12">
- <img src="img/simd.png" alt="SIMD" class="img-responsive" />
- </div>
- </div>
-
- <div class="row">
- <div class="col-xs-12">
- <h2>Advantages of a Common Data Layer</h2>
- </div>
- <div class="col-lg-6 col-lg-offset-0 col-sm-8 col-sm-offset-2 col-xs-10 col-xs-offset-1">
- <img src="img/copy.png" alt="common data layer" class="img-responsive" />
- <ul>
- <li>Each system has its own internal memory format</li>
- <li>70-80% computation wasted on serialization and deserialization</li>
- <li>Similar functionality implemented in multiple projects</li>
- </ul>
- </div>
- <div class="col-lg-6 col-lg-offset-0 col-sm-8 col-sm-offset-2 col-xs-10 col-xs-offset-1">
- <img src="img/shared.png" alt="common data layer" class="img-responsive" />
- <ul>
- <li>All systems utilize the same memory format</li>
- <li>No overhead for cross-system communication</li>
- <li>Projects can share functionality (eg, Parquet-to-Arrow reader)</li>
- </ul>
- </div>
- </div>
+ </header>
+
+ <div class="container p-lg-4">
+ <main role="main">
+ <div class="jumbotron">
+ <h1>Apache Arrow</h1>
+ <p class="lead">A cross-language development platform for in-memory data</p>
+ <p>
+ <a class="btn btn-lg btn-success" style="white-space: normal;" href="mailto:dev-subscribe@arrow.apache.org" role="button">Join Mailing List</a>
+ <a class="btn btn-lg btn-primary" style="white-space: normal;" href="/install/" role="button">Install (0.10.0 Release - 6 August 2018)</a>
+ </p>
+</div>
+<h5>
+ Interested in contributing?
+ <small class="text-muted">Join the <a href="http://mail-archives.apache.org/mod_mbox/arrow-dev/"><strong>mailing list</strong></a> or check out the <a href="https://cwiki.apache.org/confluence/display/ARROW"><strong>developer wiki</strong></a>.</small>
+</h5>
+<h5>
+ <a href="/blog/"><strong>See Latest News</strong></a>
+</h5>
+<p>
+ Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, Java, JavaScript, Python, and Ruby.
+</p>
+<hr />
+<div class="row">
+ <div class="col-lg-4">
+ <h2 class="mt-3">Fast</h2>
+ <p>Apache Arrow™ enables execution engines to take advantage of the latest SIMD (Single input multiple data) operations included in modern processors, for native vectorized optimization of analytical data processing. Columnar layout is optimized for data locality for better performance on modern hardware like CPUs and GPUs.</p>
+ <p>The Arrow memory format supports <strong>zero-copy reads</strong> for lightning-fast data access without serialization overhead.</p>
+ </div>
+ <div class="col-lg-4">
+ <h2 class="mt-3">Flexible</h2>
+ <p>Arrow acts as a new high-performance interface between various systems. It is also focused on supporting a wide variety of industry-standard programming languages. Java, C, C++, Python, Ruby, and JavaScript implementations are in progress and more languages are welcome.
+ </p>
+ </div>
+ <div class="col-lg-4">
+ <h2 class="mt-3">Standard</h2>
+ <p>Apache Arrow is backed by key developers of 13 major open source projects, including Calcite, Cassandra, Drill, Hadoop, HBase, Ibis, Impala, Kudu, Pandas, Parquet, Phoenix, Spark, and Storm making it the de-facto standard for columnar in-memory analytics.</p>
+ <p>Learn more about projects that are <a href="/powered_by/">Powered By Apache Arrow</a></p>
+ </div>
+</div>
+<hr />
+<div class="row">
+ <div class="col-md-6">
+ <h2>Performance Advantage of Columnar In-Memory</h2>
+ <p class="lead">
+ Columnar memory layout allows applications to avoid unnecessary IO and
+ accelerate analytical processing performance on modern CPUs and GPUs.
+ </p>
+ </div>
+ <div class="col-md-6">
+ <img src="/img/simd.png" alt="SIMD" class="img-fluid mx-auto" />
+ </div>
+</div>
+<hr />
+<h2>Advantages of a Common Data Layer</h2>
+<div class="row">
+ <div class="col-md-6">
+ <img src="img/copy.png" alt="common data layer" class="img-fluid mx-auto" />
+ <ul>
+ <li>Each system has its own internal memory format</li>
+ <li>70-80% computation wasted on serialization and deserialization</li>
+ <li>Similar functionality implemented in multiple projects</li>
+ </ul>
+ </div>
+ <div class="col-md-6">
+ <img src="img/shared.png" alt="common data layer" class="img-fluid mx-auto" />
+ <ul>
+ <li>All systems utilize the same memory format</li>
+ <li>No overhead for cross-system communication</li>
+ <li>Projects can share functionality (eg, Parquet-to-Arrow reader)</li>
+ </ul>
+ </div>
</div>
-<!-- /container -->
-
-</body>
-</html>
+ </main>
<hr/>
<footer class="footer">
<p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
<p>© 2017 Apache Software Foundation</p>
+ <script src="/assets/main-8d2a359fd27a888246eb638b36a4e8b68ac65b9f11c48b9fac601fa0c9a7d796.js" integrity="sha256-jSo1n9J6iIJG62OLNqTotorGW58RxIufrGAfoMmn15Y=" crossorigin="anonymous" type="text/javascript"></script>
</footer>
</div>