You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2020/05/04 17:49:59 UTC
[arrow-site] branch asf-site updated: Updating built site (build
b4479f80eb828ff887162f3c4c2782009e4590d6)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7e92136 Updating built site (build b4479f80eb828ff887162f3c4c2782009e4590d6)
7e92136 is described below
commit 7e9213616239623b4bb721872b7b6770d1b08bab
Author: Antoine Pitrou <an...@python.org>
AuthorDate: Mon May 4 17:49:49 2020 +0000
Updating built site (build b4479f80eb828ff887162f3c4c2782009e4590d6)
---
...manifest-17366d7dfa9d48774f791f0cdef11060.json} | 2 +-
.../introducing-arrow-c-data-interface/index.html | 387 +++++++++++++++++++++
blog/index.html | 15 +
feed.xml | 300 ++++++++--------
4 files changed, 540 insertions(+), 164 deletions(-)
diff --git a/assets/.sprockets-manifest-fbac4d613ea51a48907a4f64a2795c3e.json b/assets/.sprockets-manifest-17366d7dfa9d48774f791f0cdef11060.json
similarity index 63%
rename from assets/.sprockets-manifest-fbac4d613ea51a48907a4f64a2795c3e.json
rename to assets/.sprockets-manifest-17366d7dfa9d48774f791f0cdef11060.json
index 8a4b288..3b13f08 100644
--- a/assets/.sprockets-manifest-fbac4d613ea51a48907a4f64a2795c3e.json
+++ b/assets/.sprockets-manifest-17366d7dfa9d48774f791f0cdef11060.json
@@ -1 +1 @@
-{"files":{"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js":{"logical_path":"main.js","mtime":"2020-04-23T17:56:03-04:00","size":124531,"digest":"18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33","integrity":"sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM="}},"assets":{"main.js":"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js"}}
\ No newline at end of file
+{"files":{"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js":{"logical_path":"main.js","mtime":"2020-05-04T13:49:40-04:00","size":124531,"digest":"18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33","integrity":"sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM="}},"assets":{"main.js":"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js"}}
\ No newline at end of file
diff --git a/blog/2020/05/03/introducing-arrow-c-data-interface/index.html b/blog/2020/05/03/introducing-arrow-c-data-interface/index.html
new file mode 100644
index 0000000..776373d
--- /dev/null
+++ b/blog/2020/05/03/introducing-arrow-c-data-interface/index.html
@@ -0,0 +1,387 @@
+<!DOCTYPE html>
+<html lang="en-US">
+ <head>
+ <meta charset="UTF-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above meta tags *must* come first in the head; any other head content must come *after* these tags -->
+
+ <title>Introducing the Apache Arrow C Data Interface | Apache Arrow</title>
+
+
+ <!-- Begin Jekyll SEO tag v2.6.1 -->
+<meta name="generator" content="Jekyll v3.8.4" />
+<meta property="og:title" content="Introducing the Apache Arrow C Data Interface" />
+<meta name="author" content="apitrou" />
+<meta property="og:locale" content="en_US" />
+<meta name="description" content="This post introduces the Arrow C Data Interface, a simple C-based interoperability standard to simplify interactions between independent users and implementors of the Arrow in-memory format." />
+<meta property="og:description" content="This post introduces the Arrow C Data Interface, a simple C-based interoperability standard to simplify interactions between independent users and implementors of the Arrow in-memory format." />
+<link rel="canonical" href="https://arrow.apache.org/blog/2020/05/03/introducing-arrow-c-data-interface/" />
+<meta property="og:url" content="https://arrow.apache.org/blog/2020/05/03/introducing-arrow-c-data-interface/" />
+<meta property="og:site_name" content="Apache Arrow" />
+<meta property="og:image" content="https://arrow.apache.org/img/arrow.png" />
+<meta property="og:type" content="article" />
+<meta property="article:published_time" content="2020-05-03T19:00:00-04:00" />
+<meta name="twitter:card" content="summary_large_image" />
+<meta property="twitter:image" content="https://arrow.apache.org/img/arrow.png" />
+<meta property="twitter:title" content="Introducing the Apache Arrow C Data Interface" />
+<meta name="twitter:site" content="@ApacheArrow" />
+<meta name="twitter:creator" content="@apitrou" />
+<script type="application/ld+json">
+{"headline":"Introducing the Apache Arrow C Data Interface","dateModified":"2020-05-03T19:00:00-04:00","datePublished":"2020-05-03T19:00:00-04:00","publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://arrow.apache.org/img/logo.png"},"name":"apitrou"},"@type":"BlogPosting","mainEntityOfPage":{"@type":"WebPage","@id":"https://arrow.apache.org/blog/2020/05/03/introducing-arrow-c-data-interface/"},"description":"This post introduces the Arrow C Data Interface, a si [...]
+<!-- End Jekyll SEO tag -->
+
+
+ <!-- favicons -->
+ <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16.png" id="light1">
+ <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32.png" id="light2">
+ <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon.png" id="light3">
+ <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120.png" id="light4">
+ <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76.png" id="light5">
+ <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60.png" id="light6">
+ <!-- dark mode favicons -->
+ <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16-dark.png" id="dark1">
+ <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32-dark.png" id="dark2">
+ <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon-dark.png" id="dark3">
+ <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120-dark.png" id="dark4">
+ <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76-dark.png" id="dark5">
+ <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60-dark.png" id="dark6">
+
+ <script>
+ // Switch to the dark-mode favicons if prefers-color-scheme: dark
+ function onUpdate() {
+ light1 = document.querySelector('link#light1');
+ light2 = document.querySelector('link#light2');
+ light3 = document.querySelector('link#light3');
+ light4 = document.querySelector('link#light4');
+ light5 = document.querySelector('link#light5');
+ light6 = document.querySelector('link#light6');
+
+ dark1 = document.querySelector('link#dark1');
+ dark2 = document.querySelector('link#dark2');
+ dark3 = document.querySelector('link#dark3');
+ dark4 = document.querySelector('link#dark4');
+ dark5 = document.querySelector('link#dark5');
+ dark6 = document.querySelector('link#dark6');
+
+ if (matcher.matches) {
+ light1.remove();
+ light2.remove();
+ light3.remove();
+ light4.remove();
+ light5.remove();
+ light6.remove();
+ document.head.append(dark1);
+ document.head.append(dark2);
+ document.head.append(dark3);
+ document.head.append(dark4);
+ document.head.append(dark5);
+ document.head.append(dark6);
+ } else {
+ dark1.remove();
+ dark2.remove();
+ dark3.remove();
+ dark4.remove();
+ dark5.remove();
+ dark6.remove();
+ document.head.append(light1);
+ document.head.append(light2);
+ document.head.append(light3);
+ document.head.append(light4);
+ document.head.append(light5);
+ document.head.append(light6);
+ }
+ }
+ matcher = window.matchMedia('(prefers-color-scheme: dark)');
+ matcher.addListener(onUpdate);
+ onUpdate();
+ </script>
+
+ <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic,900">
+
+ <link href="/css/main.css" rel="stylesheet">
+ <link href="/css/syntax.css" rel="stylesheet">
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
+
+ <!-- Global Site Tag (gtag.js) - Google Analytics -->
+<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
+<script>
+ window.dataLayer = window.dataLayer || [];
+ function gtag(){dataLayer.push(arguments)};
+ gtag('js', new Date());
+
+ gtag('config', 'UA-107500873-1');
+</script>
+
+
+ </head>
+
+
+<body class="wrap">
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+ <a class="navbar-brand" href="/"><img src="/img/arrow-inverse-300px.png" height="60px"/></a>
+ <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#arrow-navbar" aria-controls="arrow-navbar" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <div class="collapse navbar-collapse" id="arrow-navbar">
+ <ul class="nav navbar-nav">
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownProjectLinks" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Project Links
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownProjectLinks">
+ <a class="dropdown-item" href="/install/">Installation</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="/faq/">FAQ</a>
+ <a class="dropdown-item" href="/blog/">Blog</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-user/">User Mailing List</a>
+ <a class="dropdown-item" href="http://mail-archives.apache.org/mod_mbox/arrow-dev/">Dev Mailing List</a>
+ <a class="dropdown-item" href="https://cwiki.apache.org/confluence/display/ARROW">Developer Wiki</a>
+ <a class="dropdown-item" href="/committers/">Committers</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ </div>
+ </li>
+ <li class="nav-item">
+ <a class="nav-link" href="/docs/format/Columnar.html"
+ role="button" aria-haspopup="true" aria-expanded="false">
+ Specification
+ </a>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs">Project Docs</a>
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/cpp">C++</a>
+ <a class="dropdown-item" href="/docs/java">Java</a>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib</a>
+ <a class="dropdown-item" href="/docs/js">JavaScript</a>
+ <a class="dropdown-item" href="/docs/r">R</a>
+ </div>
+ </li>
+ <!-- <li><a href="/blog">Blog</a></li> -->
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
+ </li>
+ </ul>
+ <div class="flex-row justify-content-end ml-md-auto">
+ <a class="d-sm-none d-md-inline pr-2" href="https://www.apache.org/events/current-event.html">
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ <a href="http://www.apache.org/">
+ <img src="/img/asf_logo.svg" width="120px"/>
+ </a>
+ </div>
+ </div><!-- /.navbar-collapse -->
+ </div>
+ </nav>
+
+ </header>
+
+ <div class="container p-lg-4">
+ <main role="main">
+
+
+
+<h1>
+ Introducing the Apache Arrow C Data Interface
+</h1>
+
+
+
+<p>
+ <span class="badge badge-secondary">Published</span>
+ <span class="published">
+ 03 May 2020
+ </span>
+ <br />
+ <span class="badge badge-secondary">By</span>
+
+ <a href="http://github.com/pitrou">Antoine Pitrou (apitrou) </a>
+
+
+
+</p>
+
+
+ <!--
+
+-->
+
+<p>Apache Arrow includes a cross-language, platform-independent in-memory
+<a href="https://arrow.apache.org/docs/format/Columnar.html">columnar format</a>
+allowing zero-copy data sharing and transfer between heterogenous runtimes
+and applications.</p>
+
+<p>The easiest way to use the Arrow columnar format has always been to depend
+on one of the concrete implementations developed by the Apache Arrow community.
+The project codebase contains libraries for 11 different programming languages
+so far, and will likely grow to include more languages in the future.</p>
+
+<p>However, some projects may wish to import and export the Arrow columnar format
+without taking on a new library dependency, such as the Arrow C++ library.
+We have therefore designed an alternative which exchanges data at the C level,
+conforming to a simple data definition. The C Data Interface carries no dependencies
+except a shared C ABI between binaries which use it. C ABIs are platform-wide standards
+which are necessarily adhered to by all compilers which generate binaries and are extremely
+stable, ensuring portability of libraries and executable binaries. Two libraries that utilize
+the C structures defined by the C Data Interface can do zero-copy data
+transfers at runtime without any build-time or link-time dependency
+requirements.</p>
+
+<p>The best way to learn about the C Data Interface is to read the
+<a href="https://arrow.apache.org/docs/format/CDataInterface.html">spec</a>.
+However, we will quickly go over its strong points.</p>
+
+<h2 id="two-simple-struct-definitions">Two simple struct definitions</h2>
+
+<p>To interact with the C Data Interface at the C or C++ level, the only
+thing you have to include in your code is two struct type declarations
+(and a couple of <code class="highlighter-rouge">#define</code>s for constant values). Those declarations
+only depend on standard C types, and can simply be pasted in a header
+file. Other languages can also participate as long as they provide a
+Foreign Function Interface layer; this is the case for most modern
+languages, such as Python (with <code class="highlighter-rouge">ctypes</code> or <code class="highlighter-rouge">cffi</code>), Julia, Rust, Go, etc.</p>
+
+<h2 id="zero-copy-data-sharing">Zero-copy data sharing</h2>
+
+<p>The C Data Interface passes Arrow data buffers through memory pointers. So,
+by construction, it allows you to share data from one runtime to
+another without copying it. Since the data is in standard
+<a href="https://arrow.apache.org/docs/format/Columnar.html">Arrow in-memory format</a>,
+its layout is well-defined and unambiguous.</p>
+
+<p>This design also restricts the C Data Interface to <em>in-process</em> data sharing.
+For interprocess communication, we recommend use of the Arrow
+<a href="https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc">IPC format</a>.</p>
+
+<h2 id="reduced-marshalling">Reduced marshalling</h2>
+
+<p>The C Data Interface stays close to the natural way of expressing Arrow-like
+data in C or C++. Only two aspects involve non-trivial marshalling:</p>
+
+<ul>
+ <li>the encoding of data types, using a very simple string-based language</li>
+ <li>the encoding of optional metadata, using a very simple length-prefixed format</li>
+</ul>
+
+<h2 id="separate-type-and-data-representation">Separate type and data representation</h2>
+
+<p>For applications which produce many instances of data of a single datatype
+(for example, as a stream of record batches), repeatedly reconstructing the
+datatype from its string encoding would represent unnecessary overhead. To
+address this use case, the C Data Interface defines two independent structures:
+one representing a datatype (and optional metadata), one representing a piece
+of data.</p>
+
+<h2 id="lifetime-handling">Lifetime handling</h2>
+
+<p>One common difficulty of data sharing between heterogenous runtimes is to
+correctly handle the lifetime of data. The C Data Interface allows the producer
+to define its own memory management scheme through a release callback.
+This is a simple function pointer which consumers will call when they are
+finished using the data. For example when used as a producer the Arrow C++
+library passes a release callback which simply decrements a <code class="highlighter-rouge">shared_ptr</code>’s
+reference count.</p>
+
+<h2 id="application-passing-data-between-r-and-python">Application: passing data between R and Python</h2>
+
+<p>The R and Python Arrow libraries are both based on the Arrow C++ library,
+however their respective toolchains (mandated by the R and Python packaging
+standards) are ABI-incompatible. It is therefore impossible to pass data
+directly at the C++ level between the R and Python bindings.</p>
+
+<p>Using the C Data Interface, we have circumvented this restriction and provide
+a zero-copy data sharing API between R and Python. It is based on the R
+<a href="https://rstudio.github.io/reticulate/"><code class="highlighter-rouge">reticulate</code></a> library.</p>
+
+<p>Here is an example session mixing R and Python library calls:</p>
+
+<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">library</span><span class="p">(</span><span class="n">arrow</span><span class="p">)</span><span class="w">
+</span><span class="n">library</span><span class="p">(</span><span class="n">reticulate</span><span class="p">)</span><span class="w">
+</span><span class="n">use_virtualenv</span><span class="p">(</span><span class="s2">"arrow"</span><span class="p">)</span><span class="w">
+</span><span class="n">pa</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">import</span><span class="p">(</span><span class="s2">"pyarrow"</span><span class="p">)</span><span class="w">
+
+</span><span class="c1"># Create an array in PyArrow</span><span class="w">
+</span><span class="n">a</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pa</span><span class="o">$</span><span class="n">array</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">))</span><span class="w">
+</span><span class="n">a</span><span class="w">
+
+</span><span class="c1">## Array</span><span class="w">
+</span><span class="c1">## <double></span><span class="w">
+</span><span class="c1">## [</span><span class="w">
+</span><span class="c1">## 1,</span><span class="w">
+</span><span class="c1">## 2,</span><span class="w">
+</span><span class="c1">## 3</span><span class="w">
+</span><span class="c1">## ]</span><span class="w">
+
+</span><span class="c1"># Apply R methods on the PyArrow-created array:</span><span class="w">
+</span><span class="n">a</span><span class="p">[</span><span class="n">a</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="m">1</span><span class="p">]</span><span class="w">
+
+</span><span class="c1">## Array</span><span class="w">
+</span><span class="c1">## <double></span><span class="w">
+</span><span class="c1">## [</span><span class="w">
+</span><span class="c1">## 2,</span><span class="w">
+</span><span class="c1">## 3</span><span class="w">
+</span><span class="c1">## ]</span><span class="w">
+
+</span><span class="c1"># Create an array in R and pass it to PyArrow</span><span class="w">
+</span><span class="n">b</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">Array</span><span class="o">$</span><span class="n">create</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="m">5</span><span class="p">,</span><span class="w"> </span><span class="m">6</span><span class="p">,</span><span class="w"> </span><span class="m">7</span><span class="p">))</span><span class="w">
+</span><span class="n">a_and_b</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pa</span><span class="o">$</span><span class="n">concat_arrays</span><span class="p">(</span><span class="n">r_to_py</span><span class="p">(</span><span class="nf">list</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">)))</span><span class="w">
+</span><span class="n">a_and_b</span><span class="w">
+
+</span><span class="c1">## Array</span><span class="w">
+</span><span class="c1">## <double></span><span class="w">
+</span><span class="c1">## [</span><span class="w">
+</span><span class="c1">## 1,</span><span class="w">
+</span><span class="c1">## 2,</span><span class="w">
+</span><span class="c1">## 3,</span><span class="w">
+</span><span class="c1">## 5,</span><span class="w">
+</span><span class="c1">## 6,</span><span class="w">
+</span><span class="c1">## 7</span><span class="w">
+</span><span class="c1">## ]</span><span class="w">
+</span></code></pre></div></div>
+
+ </main>
+
+ <hr/>
+<footer class="footer">
+ <p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
+ <p>© 2016-2019 The Apache Software Foundation</p>
+ <script integrity="sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM=" crossorigin="anonymous" type="text/javascript" src="/assets/main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js"></script>
+</footer>
+
+ </div>
+</body>
+</html>
diff --git a/blog/index.html b/blog/index.html
index 1056b3a..c51c96f 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -217,6 +217,21 @@
<p>
<h3>
+ <a href="/blog/2020/05/03/introducing-arrow-c-data-interface/">Introducing the Apache Arrow C Data Interface</a>
+ </h3>
+
+ <p>
+ <span class="blog-list-date">
+ 3 May 2020
+ </span>
+ </p>
+ Apache Arrow includes a cross-language, platform-independent in-memory columnar format allowing zero-copy data sharing and transfer between heterogenous runtimes and applications. The easiest way to use the Arrow columnar format has always been to depend on one of the concrete implementations developed by the Apache Arrow community. The project codebase contains...
+ </p>
+
+
+
+ <p>
+ <h3>
<a href="/blog/2020/04/21/0.17.0-release/">Apache Arrow 0.17.0 Release</a>
</h3>
diff --git a/feed.xml b/feed.xml
index 1fe1316..5473ed5 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,4 +1,139 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.8.4">Jekyll</generator><link href="https://arrow.apache.org/feed.xml" rel="self" type="application/atom+xml" /><link href="https://arrow.apache.org/" rel="alternate" type="text/html" /><updated>2020-04-23T17:55:54-04:00</updated><id>https://arrow.apache.org/feed.xml</id><title type="html">Apache Arrow</title><subtitle>Apache Arrow is a cross-language developm [...]
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.8.4">Jekyll</generator><link href="https://arrow.apache.org/feed.xml" rel="self" type="application/atom+xml" /><link href="https://arrow.apache.org/" rel="alternate" type="text/html" /><updated>2020-05-04T13:49:32-04:00</updated><id>https://arrow.apache.org/feed.xml</id><title type="html">Apache Arrow</title><subtitle>Apache Arrow is a cross-language developm [...]
+
+-->
+
+<p>Apache Arrow includes a cross-language, platform-independent in-memory
+<a href="https://arrow.apache.org/docs/format/Columnar.html">columnar format</a>
+allowing zero-copy data sharing and transfer between heterogenous runtimes
+and applications.</p>
+
+<p>The easiest way to use the Arrow columnar format has always been to depend
+on one of the concrete implementations developed by the Apache Arrow community.
+The project codebase contains libraries for 11 different programming languages
+so far, and will likely grow to include more languages in the future.</p>
+
+<p>However, some projects may wish to import and export the Arrow columnar format
+without taking on a new library dependency, such as the Arrow C++ library.
+We have therefore designed an alternative which exchanges data at the C level,
+conforming to a simple data definition. The C Data Interface carries no dependencies
+except a shared C ABI between binaries which use it. C ABIs are platform-wide standards
+which are necessarily adhered to by all compilers which generate binaries and are extremely
+stable, ensuring portability of libraries and executable binaries. Two libraries that utilize
+the C structures defined by the C Data Interface can do zero-copy data
+transfers at runtime without any build-time or link-time dependency
+requirements.</p>
+
+<p>The best way to learn about the C Data Interface is to read the
+<a href="https://arrow.apache.org/docs/format/CDataInterface.html">spec</a>.
+However, we will quickly go over its strong points.</p>
+
+<h2 id="two-simple-struct-definitions">Two simple struct definitions</h2>
+
+<p>To interact with the C Data Interface at the C or C++ level, the only
+thing you have to include in your code is two struct type declarations
+(and a couple of <code class="highlighter-rouge">#define</code>s for constant values). Those declarations
+only depend on standard C types, and can simply be pasted in a header
+file. Other languages can also participate as long as they provide a
+Foreign Function Interface layer; this is the case for most modern
+languages, such as Python (with <code class="highlighter-rouge">ctypes</code> or <code class="highlighter-rouge">cffi</code>), Julia, Rust, Go, etc.</p>
+
+<h2 id="zero-copy-data-sharing">Zero-copy data sharing</h2>
+
+<p>The C Data Interface passes Arrow data buffers through memory pointers. So,
+by construction, it allows you to share data from one runtime to
+another without copying it. Since the data is in standard
+<a href="https://arrow.apache.org/docs/format/Columnar.html">Arrow in-memory format</a>,
+its layout is well-defined and unambiguous.</p>
+
+<p>This design also restricts the C Data Interface to <em>in-process</em> data sharing.
+For interprocess communication, we recommend use of the Arrow
+<a href="https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc">IPC format</a>.</p>
+
+<h2 id="reduced-marshalling">Reduced marshalling</h2>
+
+<p>The C Data Interface stays close to the natural way of expressing Arrow-like
+data in C or C++. Only two aspects involve non-trivial marshalling:</p>
+
+<ul>
+ <li>the encoding of data types, using a very simple string-based language</li>
+ <li>the encoding of optional metadata, using a very simple length-prefixed format</li>
+</ul>
+
+<h2 id="separate-type-and-data-representation">Separate type and data representation</h2>
+
+<p>For applications which produce many instances of data of a single datatype
+(for example, as a stream of record batches), repeatedly reconstructing the
+datatype from its string encoding would represent unnecessary overhead. To
+address this use case, the C Data Interface defines two independent structures:
+one representing a datatype (and optional metadata), one representing a piece
+of data.</p>
+
+<h2 id="lifetime-handling">Lifetime handling</h2>
+
+<p>One common difficulty of data sharing between heterogenous runtimes is to
+correctly handle the lifetime of data. The C Data Interface allows the producer
+to define its own memory management scheme through a release callback.
+This is a simple function pointer which consumers will call when they are
+finished using the data. For example when used as a producer the Arrow C++
+library passes a release callback which simply decrements a <code class="highlighter-rouge">shared_ptr</code>’s
+reference count.</p>
+
+<h2 id="application-passing-data-between-r-and-python">Application: passing data between R and Python</h2>
+
+<p>The R and Python Arrow libraries are both based on the Arrow C++ library,
+however their respective toolchains (mandated by the R and Python packaging
+standards) are ABI-incompatible. It is therefore impossible to pass data
+directly at the C++ level between the R and Python bindings.</p>
+
+<p>Using the C Data Interface, we have circumvented this restriction and provide
+a zero-copy data sharing API between R and Python. It is based on the R
+<a href="https://rstudio.github.io/reticulate/"><code class="highlighter-rouge">reticulate</code></a> library.</p>
+
+<p>Here is an example session mixing R and Python library calls:</p>
+
+<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">library</span><span class="p">(</span><span class="n">arrow</span><span class="p">)</span><span class="w">
+</span><span class="n">library</span><span class="p">(</span><span class="n">reticulate</span><span class="p">)</span><span class="w">
+</span><span class="n">use_virtualenv</span><span class="p">(</span><span class="s2">"arrow"</span><span class="p">)</span><span class="w">
+</span><span class="n">pa</span><span class="w"> </span><span class="o">&lt;-</span><span class="w"> </span><span class="n">import</span><span class="p">(</span><span class="s2">"pyarrow"</span><span class="p">)</span><span class="w">
+
+</span><span class="c1"># Create an array in PyArrow</span><span class="w">
+</span><span class="n">a</span><span class="w"> </span><span class="o">&lt;-</span><span class="w"> </span><span class="n">pa</span><span class="o">$</span><span class="n">array</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><s [...]
+</span><span class="n">a</span><span class="w">
+
+</span><span class="c1">## Array</span><span class="w">
+</span><span class="c1">## &lt;double&gt;</span><span class="w">
+</span><span class="c1">## [</span><span class="w">
+</span><span class="c1">## 1,</span><span class="w">
+</span><span class="c1">## 2,</span><span class="w">
+</span><span class="c1">## 3</span><span class="w">
+</span><span class="c1">## ]</span><span class="w">
+
+</span><span class="c1"># Apply R methods on the PyArrow-created array:</span><span class="w">
+</span><span class="n">a</span><span class="p">[</span><span class="n">a</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="m">1</span><span class="p">]</span><span class="w">
+
+</span><span class="c1">## Array</span><span class="w">
+</span><span class="c1">## &lt;double&gt;</span><span class="w">
+</span><span class="c1">## [</span><span class="w">
+</span><span class="c1">## 2,</span><span class="w">
+</span><span class="c1">## 3</span><span class="w">
+</span><span class="c1">## ]</span><span class="w">
+
+</span><span class="c1"># Create an array in R and pass it to PyArrow</span><span class="w">
+</span><span class="n">b</span><span class="w"> </span><span class="o">&lt;-</span><span class="w"> </span><span class="n">Array</span><span class="o">$</span><span class="n">create</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span>& [...]
+</span><span class="n">a_and_b</span><span class="w"> </span><span class="o">&lt;-</span><span class="w"> </span><span class="n">pa</span><span class="o">$</span><span class="n">concat_arrays</span><span class="p">(</span><span class="n">r_to_py</span><span class="p"> [...]
+</span><span class="n">a_and_b</span><span class="w">
+
+</span><span class="c1">## Array</span><span class="w">
+</span><span class="c1">## &lt;double&gt;</span><span class="w">
+</span><span class="c1">## [</span><span class="w">
+</span><span class="c1">## 1,</span><span class="w">
+</span><span class="c1">## 2,</span><span class="w">
+</span><span class="c1">## 3,</span><span class="w">
+</span><span class="c1">## 5,</span><span class="w">
+</span><span class="c1">## 6,</span><span class="w">
+</span><span class="c1">## 7</span><span class="w">
+</span><span class="c1">## ]</span><span class="w">
+</span></code></pre></div></div></content><author><name>apitrou</name></author><summary type="html">Apache Arrow includes a cross-language, platform-independent in-memory columnar format allowing zero-copy data sharing and transfer between heterogenous runtimes and applications. The easiest way to use the Arrow columnar format has always been to depend on one of the concrete implementations developed by the Apache Arrow community. The project codebase contai [...]
-->
@@ -1862,165 +1997,4 @@ in the Arrow binary protocol</li>
needs</li>
<li><a href="https://lists.apache.org/thread.html/a99124e57c14c3c9ef9d98f3c80cfe1dd25496bf3ff7046778add937@%3Cdev.arrow.apache.org%3E">Sparse representation and compression in Arrow</a></li>
<li><a href="https://lists.apache.org/thread.html/82a7c026ad18dbe9fdbcffa3560979aff6fd86dd56a49f40d9cfb46e@%3Cdev.arrow.apache.org%3E">Flight extensions: middleware API and generalized Put operations</a></li>
-</ul></content><author><name>pmc</name></author><summary type="html">The Apache Arrow team is pleased to announce the 0.14.0 release. This covers 3 months of development work and includes 602 resolved issues from 75 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The complete changelog is also available. This post will give some brief highlights in the project since the 0.13.0 release from April. New committers Since the 0.13.0 relea [...]
-
--->
-
-<p>The Apache Arrow team is pleased to announce the 0.13.0 release. This covers
-more than 2 months of development work and includes <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%200.13.0"><strong>550 resolved
-issues</strong></a> from <a href="https://arrow.apache.org/release/0.13.0.html#contributors"><strong>81 distinct contributors</strong></a>.</p>
-
-<p>See the <a href="https://arrow.apache.org/install">Install Page</a> to learn how to get the libraries for your
-platform. The <a href="https://arrow.apache.org/release/0.13.0.html">complete changelog</a> is also available.</p>
-
-<p>While it’s a large release, this post will give some brief highlights in the
-project since the 0.12.0 release from January.</p>
-
-<h2 id="new-committers-and-pmc-member">New committers and PMC member</h2>
-
-<p>The Arrow team is growing! Since the 0.12.0 release we have increased the size
-of our committer and PMC rosters.</p>
-
-<ul>
- <li><a href="https://github.com/andygrove">Andy Grove</a> was promoted to PMC member</li>
- <li><a href="https://github.com/paddyhoran">Paddy Horan</a> was added as a committer</li>
- <li><a href="https://github.com/emkornfield">Micah Kornfield</a> was added as a committer</li>
- <li><a href="https://github.com/pravindra">Ravindra Pindikura</a> was added as a committer</li>
- <li><a href="https://github.com/sunchao">Chao Sun</a> was added as a committer</li>
-</ul>
-
-<p>Thank you for all your contributions!</p>
-
-<h2 id="rust-datafusion-query-engine-donation">Rust DataFusion Query Engine donation</h2>
-
-<p>Since the last release, we received a donation of <a href="http://incubator.apache.org/ip-clearance/arrow-rust-datafusion.html">DataFusion</a>, a
-Rust-native query engine for the Arrow columnar format, whose development had
-been led prior by Andy Grove. <a href="http://arrow.apache.org/blog/2019/02/04/datafusion-donation/">Read more about DataFusion</a> in our February
-blog post.</p>
-
-<p>This is an exciting development for the Rust community, and we look forward to
-developing more analytical query processing within the Apache Arrow project.</p>
-
-<h2 id="arrow-flight-grpc-progress">Arrow Flight gRPC progress</h2>
-
-<p>Over the last couple months, we have made significant progress on Arrow Flight,
-an Arrow-native data messaging framework. We have integration tests to check
-C++ and Java compatibility, and we have added Python bindings for the C++
-library. We will write a future blog post to go into more detail about how
-Flight works.</p>
-
-<h2 id="c-notes">C++ notes</h2>
-
-<p>There were 231 issues relating to C++ in this release, far too much to
-summarize in a blog post. Some notable items include:</p>
-
-<ul>
- <li>An experimental <code class="highlighter-rouge">ExtensionType</code> was developed for creating user-defined data
-types that can be embedded in the Arrow binary protocol. This is not yet
-finalized, but <a href="https://github.com/apache/arrow/blob/master/cpp/src/arrow/extension_type.h">feedback would be welcome</a>.</li>
- <li>We have undertaken a significant reworking of our CMake build system for C++
-to make the third party dependencies more configurable. Among other things,
-this eases work on packaging for Linux distributions. Read more about this in
-the <a href="https://github.com/apache/arrow/blob/master/docs/source/developers/cpp.rst#build-dependency-management">C++ developer documentation</a>.</li>
- <li>Laying more groundwork for an Arrow-native in-memory query engine</li>
- <li>We began building a reader for line-delimited JSON files</li>
- <li>Gandiva can now be compiled on Windows with Visual Studio</li>
-</ul>
-
-<h2 id="c-notes-1">C# Notes</h2>
-
-<p>C# .NET development has picked up since the initial code donation last
-fall. 11 issues were resolved this release cycle.</p>
-
-<p>The Arrow C# package is <a href="https://www.nuget.org/packages/Apache.Arrow/0.13.0">now available via NuGet</a>.</p>
-
-<h2 id="go-notes">Go notes</h2>
-
-<p>8 Go-related issues were resolved. A notable feature is the addition of a CSV
-file writer.</p>
-
-<h2 id="java-notes">Java notes</h2>
-
-<p>26 Java issues were resolved. Outside of Flight-related work, some notable
-items include:</p>
-
-<ul>
- <li>Migration to Java 8 date and time APIs from Joda</li>
- <li>Array type support in JDBC adapter</li>
-</ul>
-
-<h2 id="javascript-notes">Javascript Notes</h2>
-
-<p>The recent <a href="https://www.npmjs.com/package/apache-arrow/v/0.4.1">JavaScript 0.4.1 release</a> is the last JavaScript-only release
-of Apache Arrow. Starting with 0.13 the Javascript implementation is now
-included in mainline Arrow releases! The version number of the released
-JavaScript packages will now be in sync with the mainline version number.</p>
-
-<h2 id="python-notes">Python notes</h2>
-
-<p>86 Python-related issues were resolved. Some highlights include:</p>
-
-<ul>
- <li>The Gandiva LLVM expression compiler is now available in the Python wheels
-through the <code class="highlighter-rouge">pyarrow.gandiva</code> module.</li>
- <li>Flight RPC bindings</li>
- <li>Improved pandas serialization performance with RangeIndex</li>
- <li>pyarrow can be used without pandas installed</li>
-</ul>
-
-<p>Note that Apache Arrow will continue to support Python 2.7 until January 2020.</p>
-
-<h2 id="ruby-and-c-glib-notes">Ruby and C GLib notes</h2>
-
-<p>36 C/GLib- and Ruby-related issues were resolved. The work continues to follow
-the upstream work in the C++ project.</p>
-
-<ul>
- <li><code class="highlighter-rouge">Arrow::RecordBatch#raw_records</code> was added. It can convert a record batch to
-a Ruby’s array in 10x-200x faster than the same conversion by a pure-Ruby
-implementation.</li>
-</ul>
-
-<h2 id="rust-notes">Rust notes</h2>
-
-<p>69 Rust-related issues were resolved. Many of these relate to ongoing work in
-the DataFusion query engine. Some notable items include:</p>
-
-<ul>
- <li>Date/time support</li>
- <li>SIMD for arithmetic operations</li>
- <li>Writing CSV and reading line-delimited JSON</li>
- <li>Parquet data source support for DataFusion</li>
- <li>Prototype DataFrame-style API for DataFusion</li>
- <li>Continued evolution of Parquet file reader</li>
-</ul>
-
-<h2 id="r-development-progress">R development progress</h2>
-
-<p>The Arrow R developers have expanded the scope of the R language bindings and
-additionally worked on packaging support to be able to submit the package to
-CRAN in the near future. 23 issues were resolved for this release.</p>
-
-<p><a href="http://arrow.apache.org/blog/2019/01/25/r-spark-improvements/">We wrote in January about ongoing work</a> to accelerate R work on Apache Spark
-using Arrow.</p>
-
-<h2 id="community-discussions-ongoing">Community Discussions Ongoing</h2>
-
-<p>There are a number of active discussions ongoing on the developer
-<code class="highlighter-rouge">dev@arrow.apache.org</code> mailing list. We look forward to hearing from the
-community there:</p>
-
-<ul>
- <li><strong>Benchmarking</strong>: we are working to create tools for tracking all of our
-benchmark results on a commit-by-commit basis in a centralized database
-schema so that we can monitor for performance regressions over time. We hope
-to develop a publicly viewable benchmark result dashboard.</li>
- <li><strong>C++ Datasets</strong>: development of a unified API for reading and writing
-datasets stored in various common formats like Parquet, JSON, and CSV.</li>
- <li><strong>C++ Query Engine</strong>: architecture of a parallel Arrow-native query engine
-for C++</li>
- <li><strong>Arrow Flight Evolution</strong>: adding features to support different real-world
-data messaging use cases</li>
- <li><strong>Arrow Columnar Format evolution</strong>: we are discussing a new “duration” or
-“time interval” type and some other additions to the Arrow columnar format.</li>
-</ul></content><author><name>wesm</name></author><summary type="html">The Apache Arrow team is pleased to announce the 0.13.0 release. This covers more than 2 months of development work and includes 550 resolved issues from 81 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The complete changelog is also available. While it’s a large release, this post will give some brief highlights in the project since the 0.12.0 release from Janua [...]
\ No newline at end of file
+</ul></content><author><name>pmc</name></author><summary type="html">The Apache Arrow team is pleased to announce the 0.14.0 release. This covers 3 months of development work and includes 602 resolved issues from 75 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The complete changelog is also available. This post will give some brief highlights in the project since the 0.13.0 release from April. New committers Since the 0.13.0 relea [...]
\ No newline at end of file