You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2021/02/03 19:37:07 UTC
[arrow-site] branch asf-site updated: Updating built site (build
593109a5807dc3eb35e52d0a5201581e02f13a10)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 36a29e0 Updating built site (build 593109a5807dc3eb35e52d0a5201581e02f13a10)
36a29e0 is described below
commit 36a29e05f93b16f0a48b6bed2500f3307aa24e4a
Author: Neal Richardson <ne...@gmail.com>
AuthorDate: Wed Feb 3 19:35:55 2021 +0000
Updating built site (build 593109a5807dc3eb35e52d0a5201581e02f13a10)
---
...manifest-fb3f0e38a66ae55c491aae697b77db8f.json} | 2 +-
blog/2021/01/25/3.0.0-release/index.html | 584 ++++++++++++++++++++
blog/index.html | 15 +
docs/c_glib/index.html | 247 +++++++++
feed.xml | 589 ++++++++++++---------
5 files changed, 1174 insertions(+), 263 deletions(-)
diff --git a/assets/.sprockets-manifest-c5189537465fe1e01bcff8f9c51d0ba7.json b/assets/.sprockets-manifest-fb3f0e38a66ae55c491aae697b77db8f.json
similarity index 63%
rename from assets/.sprockets-manifest-c5189537465fe1e01bcff8f9c51d0ba7.json
rename to assets/.sprockets-manifest-fb3f0e38a66ae55c491aae697b77db8f.json
index dbb06f4..82416db 100644
--- a/assets/.sprockets-manifest-c5189537465fe1e01bcff8f9c51d0ba7.json
+++ b/assets/.sprockets-manifest-fb3f0e38a66ae55c491aae697b77db8f.json
@@ -1 +1 @@
-{"files":{"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js":{"logical_path":"main.js","mtime":"2021-01-25T20:26:18-05:00","size":124531,"digest":"18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33","integrity":"sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM="}},"assets":{"main.js":"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js"}}
\ No newline at end of file
+{"files":{"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js":{"logical_path":"main.js","mtime":"2021-02-03T14:35:43-05:00","size":124531,"digest":"18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33","integrity":"sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM="}},"assets":{"main.js":"main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js"}}
\ No newline at end of file
diff --git a/blog/2021/01/25/3.0.0-release/index.html b/blog/2021/01/25/3.0.0-release/index.html
new file mode 100644
index 0000000..04bdd6d
--- /dev/null
+++ b/blog/2021/01/25/3.0.0-release/index.html
@@ -0,0 +1,584 @@
+<!DOCTYPE html>
+<html lang="en-US">
+ <head>
+ <meta charset="UTF-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above meta tags *must* come first in the head; any other head content must come *after* these tags -->
+
+ <title>Apache Arrow 3.0.0 Release | Apache Arrow</title>
+
+
+ <!-- Begin Jekyll SEO tag v2.7.1 -->
+<meta name="generator" content="Jekyll v3.8.4" />
+<meta property="og:title" content="Apache Arrow 3.0.0 Release" />
+<meta name="author" content="pmc" />
+<meta property="og:locale" content="en_US" />
+<meta name="description" content="The Apache Arrow team is pleased to announce the 3.0.0 release. This covers over 3 months of development work and includes 666 resolved issues from 106 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The release notes below are not exhaustive and only expose selected highlights of the release. Many other bugfixes and improvements have been made: we refer you to the complete changelog. Columnar Format Notes [...]
+<meta property="og:description" content="The Apache Arrow team is pleased to announce the 3.0.0 release. This covers over 3 months of development work and includes 666 resolved issues from 106 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The release notes below are not exhaustive and only expose selected highlights of the release. Many other bugfixes and improvements have been made: we refer you to the complete changelog. Columnar Forma [...]
+<link rel="canonical" href="https://arrow.apache.org/blog/2021/01/25/3.0.0-release/" />
+<meta property="og:url" content="https://arrow.apache.org/blog/2021/01/25/3.0.0-release/" />
+<meta property="og:site_name" content="Apache Arrow" />
+<meta property="og:image" content="https://arrow.apache.org/img/arrow.png" />
+<meta property="og:type" content="article" />
+<meta property="article:published_time" content="2021-01-25T01:00:00-05:00" />
+<meta name="twitter:card" content="summary_large_image" />
+<meta property="twitter:image" content="https://arrow.apache.org/img/arrow.png" />
+<meta property="twitter:title" content="Apache Arrow 3.0.0 Release" />
+<meta name="twitter:site" content="@ApacheArrow" />
+<meta name="twitter:creator" content="@pmc" />
+<script type="application/ld+json">
+{"author":{"@type":"Person","name":"pmc"},"publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://arrow.apache.org/img/logo.png"},"name":"pmc"},"datePublished":"2021-01-25T01:00:00-05:00","description":"The Apache Arrow team is pleased to announce the 3.0.0 release. This covers over 3 months of development work and includes 666 resolved issues from 106 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The release not [...]
+<!-- End Jekyll SEO tag -->
+
+
+ <!-- favicons -->
+ <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16.png" id="light1">
+ <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32.png" id="light2">
+ <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon.png" id="light3">
+ <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120.png" id="light4">
+ <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76.png" id="light5">
+ <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60.png" id="light6">
+ <!-- dark mode favicons -->
+ <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16-dark.png" id="dark1">
+ <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32-dark.png" id="dark2">
+ <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon-dark.png" id="dark3">
+ <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120-dark.png" id="dark4">
+ <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76-dark.png" id="dark5">
+ <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60-dark.png" id="dark6">
+
+ <script>
+ // Switch to the dark-mode favicons if prefers-color-scheme: dark
+ function onUpdate() {
+ light1 = document.querySelector('link#light1');
+ light2 = document.querySelector('link#light2');
+ light3 = document.querySelector('link#light3');
+ light4 = document.querySelector('link#light4');
+ light5 = document.querySelector('link#light5');
+ light6 = document.querySelector('link#light6');
+
+ dark1 = document.querySelector('link#dark1');
+ dark2 = document.querySelector('link#dark2');
+ dark3 = document.querySelector('link#dark3');
+ dark4 = document.querySelector('link#dark4');
+ dark5 = document.querySelector('link#dark5');
+ dark6 = document.querySelector('link#dark6');
+
+ if (matcher.matches) {
+ light1.remove();
+ light2.remove();
+ light3.remove();
+ light4.remove();
+ light5.remove();
+ light6.remove();
+ document.head.append(dark1);
+ document.head.append(dark2);
+ document.head.append(dark3);
+ document.head.append(dark4);
+ document.head.append(dark5);
+ document.head.append(dark6);
+ } else {
+ dark1.remove();
+ dark2.remove();
+ dark3.remove();
+ dark4.remove();
+ dark5.remove();
+ dark6.remove();
+ document.head.append(light1);
+ document.head.append(light2);
+ document.head.append(light3);
+ document.head.append(light4);
+ document.head.append(light5);
+ document.head.append(light6);
+ }
+ }
+ matcher = window.matchMedia('(prefers-color-scheme: dark)');
+ matcher.addListener(onUpdate);
+ onUpdate();
+ </script>
+
+ <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic,900">
+
+ <link href="/css/main.css" rel="stylesheet">
+ <link href="/css/syntax.css" rel="stylesheet">
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
+
+ <!-- Global Site Tag (gtag.js) - Google Analytics -->
+<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
+<script>
+ window.dataLayer = window.dataLayer || [];
+ function gtag(){dataLayer.push(arguments)};
+ gtag('js', new Date());
+
+ gtag('config', 'UA-107500873-1');
+</script>
+
+
+ </head>
+
+
+<body class="wrap">
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+
+ <a class="navbar-brand no-padding" href="/"><img src="/img/arrow-inverse-300px.png" height="40px"/></a>
+
+ <button class="navbar-toggler ml-auto" type="button" data-toggle="collapse" data-target="#arrow-navbar" aria-controls="arrow-navbar" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <div class="collapse navbar-collapse justify-content-end" id="arrow-navbar">
+ <ul class="nav navbar-nav">
+ <li class="nav-item"><a class="nav-link" href="/overview/" role="button" aria-haspopup="true" aria-expanded="false">Overview</a></li>
+ <li class="nav-item"><a class="nav-link" href="/faq/" role="button" aria-haspopup="true" aria-expanded="false">FAQ</a></li>
+ <li class="nav-item"><a class="nav-link" href="/blog" role="button" aria-haspopup="true" aria-expanded="false">Blog</a></li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownGetArrow" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Get Arrow
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownGetArrow">
+ <a class="dropdown-item" href="/install/">Install</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs">Project Docs</a>
+ <a class="dropdown-item" href="/docs/format/Columnar.html">Format</a>
+ <hr/>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib</a>
+ <a class="dropdown-item" href="/docs/cpp">C++</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow/blob/master/csharp/README.md">C#</a>
+ <a class="dropdown-item" href="https://godoc.org/github.com/apache/arrow/go/arrow">Go</a>
+ <a class="dropdown-item" href="/docs/java">Java</a>
+ <a class="dropdown-item" href="/docs/js">JavaScript</a>
+ <a class="dropdown-item" href="https://arrow.juliadata.org/stable/">Julia</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow/blob/master/matlab/README.md">MATLAB</a>
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/r">R</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow/blob/master/ruby/README.md">Ruby</a>
+ <a class="dropdown-item" href="https://docs.rs/crate/arrow/">Rust</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="/community/">Communication</a>
+ <a class="dropdown-item" href="/docs/developers/contributing.html">Contributing</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ <a class="dropdown-item" href="/committers/">Governance</a>
+ <a class="dropdown-item" href="/use_cases/">Use Cases</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ <a class="dropdown-item" href="/security/">Security</a>
+ <a class="dropdown-item" href="https://www.apache.org/foundation/policies/conduct.html">Code of Conduct</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
+ </a>
+ <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
+ </li>
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </header>
+
+ <div class="container p-4 pt-5">
+ <div class="col-md-8 mx-auto">
+ <main role="main" class="pb-5">
+
+<h1>
+ Apache Arrow 3.0.0 Release
+</h1>
+<hr class="mt-4 mb-3">
+
+
+
+<p class="mb-4 pb-1">
+ <span class="badge badge-secondary">Published</span>
+ <span class="published mr-3">
+ 25 Jan 2021
+ </span>
+ <br />
+ <span class="badge badge-secondary">By</span>
+
+ <a class="mr-3" href="https://arrow.apache.org">The Apache Arrow PMC (pmc) </a>
+
+
+
+</p>
+
+
+ <!--
+
+-->
+
+<p>The Apache Arrow team is pleased to announce the 3.0.0 release. This covers
+over 3 months of development work and includes <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%203.0.0"><strong>666 resolved issues</strong></a>
+from <a href="/release/3.0.0.html#contributors"><strong>106 distinct contributors</strong></a>. See the Install Page to learn how to
+get the libraries for your platform.</p>
+
+<p>The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the <a href="/release/3.0.0.html">complete changelog</a>.</p>
+
+<h2 id="columnar-format-notes">Columnar Format Notes</h2>
+
+<p>The Decimal256 data type, which was already supported by the Arrow columnar
+format specification, is now implemented in C++ and Java (ARROW-9747).</p>
+
+<h2 id="arrow-flight-rpc-notes">Arrow Flight RPC notes</h2>
+
+<p>Authentication in C++/Java/Python has been overhauled, allowing more flexible authentication methods and use of standard headers.
+Support for cookies has also been added.
+The C++/Java implementations are now more permissive when parsing messages in order to interoperate better with other Flight implementations.</p>
+
+<p>A basic Flight implementation for C#/.NET has been added.
+See the <a href="https://arrow.apache.org/docs/status.html#flight-rpc">implementation status matrix</a> for details.</p>
+
+<h2 id="c-notes">C++ notes</h2>
+
+<p>The default memory pool can now be changed at runtime using the environment
+variable <code class="highlighter-rouge">ARROW_DEFAULT_MEMORY_POOL</code> (ARROW-11009). The environment variable
+is inspected at process startup. This is useful when trying to diagnose memory
+consumption issues with Arrow.</p>
+
+<p>STL-like iterators are now provided over concrete arrays. Those are useful for
+non-performance critical tasks, for example testing (ARROW-10776).</p>
+
+<p>It is now possible to concatenate dictionary arrays with unequal dictionaries.
+The dictionaries are unified when concatenating, for supported data types
+(ARROW-5336).</p>
+
+<p>Threads in a thread pool are now spawned lazily as needed for enqueued
+tasks, up to the configured capacity. They used to be spawned upfront on
+creation of the thread pool (ARROW-10038).</p>
+
+<h3 id="compute-layer">Compute layer</h3>
+
+<p>Comprehensive documentation for compute functions is now available:
+https://arrow.apache.org/docs/cpp/compute.html</p>
+
+<p>Compute functions for string processing have been added for:</p>
+
+<ul>
+ <li>splitting on whitespace (ASCII and Unicode flavors) and splitting on a
+pattern (ARROW-9991);</li>
+ <li>trimming characters (ARROW-9128).</li>
+</ul>
+
+<p>Behavior of the <code class="highlighter-rouge">index_in</code> and <code class="highlighter-rouge">is_in</code> compute functions with nulls has been
+changed for consistency (ARROW-10663).</p>
+
+<p>Multiple-column sort kernels are now available for tables and record batches
+(ARROW-8199, ARROW-10796, ARROW-10790).</p>
+
+<p>Performance of table filtering has been vastly improved (ARROW-10569).</p>
+
+<p>Scalar arguments are now accepted for more compute functions.</p>
+
+<p>Compute functions <code class="highlighter-rouge">quantile</code> (ARROW-10831) and <code class="highlighter-rouge">is_nan</code> (ARROW-11043) have been
+added for numeric data.</p>
+
+<p>Aggregation functions <code class="highlighter-rouge">any</code> (ARROW-1846) and <code class="highlighter-rouge">all</code> (ARROW-10301) have been
+added for boolean data.</p>
+
+<h3 id="dataset">Dataset</h3>
+
+<p>The <code class="highlighter-rouge">Expression</code> hierarchy has simplified to a wrapper around literals, field references,
+or calls to named functions. This enables usage of any compute function while filtering
+with no boilerplate.</p>
+
+<p>Parquet statistics are lazily parsed in <code class="highlighter-rouge">ParquetDatasetFactory</code> and
+<code class="highlighter-rouge">ParquetFileFragment</code> for shorter construction time.</p>
+
+<h3 id="csv">CSV</h3>
+
+<p>Conversion of string columns is now faster thanks to faster UTF-8 validation
+of small strings (ARROW-10313).</p>
+
+<p>Conversion of floating-point columns is now faster thanks to optimized
+string-to-double conversion routines (ARROW-10328).</p>
+
+<p>Parsing of ISO8601 timestamps is now more liberal: trailing zeros can
+be omitted in the fractional part (ARROW-10337).</p>
+
+<p>Fixed a bug where null detection could give the wrong results on some platforms
+(ARROW-11067).</p>
+
+<p>Added type inference for Date32 columns for values in the form <code class="highlighter-rouge">YYYY-MM-DD</code>
+(ARROW-11247).</p>
+
+<h3 id="feather">Feather</h3>
+
+<p>Fixed reading of compressed Feather files written with Arrow 0.17 (ARROW-11163).</p>
+
+<h3 id="filesystem-layer">Filesystem layer</h3>
+
+<p>S3 recursive tree walks now benefit from a parallel implementation, where reads
+of multiple child directories are now issued concurrently (ARROW-10788).</p>
+
+<p>Improved empty directory detection to be mindful of differences between Amazon
+and Minio S3 implementations (ARROW-10942).</p>
+
+<h3 id="flight-rpc">Flight RPC</h3>
+
+<p>IPv6 host addresses are now supported (ARROW-10475).</p>
+
+<h3 id="ipc">IPC</h3>
+
+<p>It is now possible to emit dictionary deltas where possible using the IPC
+stream writer. This is governed by a new variable in the <code class="highlighter-rouge">IpcWriteOptions</code> class
+(ARROW-6883).</p>
+
+<p>It is now possible to read wider tables, which used to fail due to reaching a
+limit during Flatbuffers verification (ARROW-10056).</p>
+
+<h3 id="parquet">Parquet</h3>
+
+<p>Fixed reading of LZ4-compressed Parquet columns emitted by the Java Parquet
+implementation (ARROW-11301).</p>
+
+<p>Fixed a bug where writing multiple batches of nullable nested strings to Parquet
+would not write any data in batches after the first one (ARROW-10493)</p>
+
+<p>The Decimal256 data type can be read from and written to Parquet (ARROW-10607).</p>
+
+<p>LargeString and LargeBinary data can now be written to Parquet (ARROW-10426).</p>
+
+<h2 id="c-notes-1">C# notes</h2>
+
+<p>The .NET package added initial support for Arrow Flight clients and servers. Support is enabled through two new NuGet packages <a href="https://www.nuget.org/packages/Apache.Arrow.Flight/">Apache.Arrow.Flight</a> (client) and <a href="https://www.nuget.org/packages/Apache.Arrow.Flight.AspNetCore/">Apache.Arrow.Flight.AspNetCore</a> (server).</p>
+
+<p>Also fixed an issue where ArrowStreamWriter wasn’t writing schema metadata to Arrow streams.</p>
+
+<h2 id="julia-notes">Julia notes</h2>
+
+<p>This is the first release to officially include
+<a href="https://github.com/apache/arrow/tree/master/julia/Arrow">an implementation</a>
+for the Julia language. The pure Julia implementation includes support
+for <a href="https://arrow.apache.org/docs/status.html">wide coverage of the format specification</a>.
+Additional details can be found in the
+<a href="https://julialang.org/blog/2021/01/arrow/">julialang.org blog post</a>.</p>
+
+<h2 id="python-notes">Python notes</h2>
+
+<p>Support for Python 3.9 was added (ARROW-10224), and support for Python 3.5
+was removed (ARROW-5679).</p>
+
+<p>Support for building manylinux1 packages has been removed (ARROW-11212).
+PyArrow continues to be available as manylinux2010 and manylinux2014 wheels.</p>
+
+<p>The minimal required version for NumPy is now 1.16.6. Note that when upgrading
+NumPy to 1.20, you also need to upgrade pyarrow to 3.0.0 to ensure compatibility,
+as this pyarrow release fixed a compatibility issue with NumPy 1.20 (ARROW-10833).</p>
+
+<p>Compute functions are now automatically exported from C++ to the <code class="highlighter-rouge">pyarrow.compute</code>
+module, and they have docstrings matching their C++ definition.</p>
+
+<p>An <code class="highlighter-rouge">iter_batches()</code> method is now available for reading a Parquet file iteratively
+(ARROW-7800).</p>
+
+<p>Alternate memory pools (such as mimalloc, jemalloc or the C malloc-based memory
+pool) are now available from Python (ARROW-11049).</p>
+
+<p>Fixed a potential deadlock when importing pandas from several threads (ARROW-10519).</p>
+
+<p>See the C++ notes above for additional details.</p>
+
+<h2 id="r-notes">R notes</h2>
+
+<p>This release contains new features for the Flight RPC wrapper, better
+support for saving R metadata (including <code class="highlighter-rouge">sf</code> spatial data) to Feather and
+Parquet files, several significant improvements to speed and memory management,
+and many other enhancements.</p>
+
+<p>For more on what’s in the 3.0.0 R package, see the <a href="/docs/r/news/">R changelog</a>.</p>
+
+<h2 id="ruby-and-c-glib-notes">Ruby and C GLib notes</h2>
+
+<h3 id="ruby">Ruby</h3>
+
+<p>In Ruby binding, 256-bit decimal support and <code class="highlighter-rouge">Arrow::FixedBinaryArrayBuilder</code> are added likewise C GLib below.</p>
+
+<h3 id="c-glib">C GLib</h3>
+
+<p>In the version 3.0.0 of C GLib consists of many new features.</p>
+
+<p>A chunked array, a record batch, and a table support <code class="highlighter-rouge">sort_indices</code> function as well as an array.
+These functions including array’s support to specify sorting option.
+<code class="highlighter-rouge">garrow_array_sort_to_indices</code> has been renamed to <code class="highlighter-rouge">garrow_array_sort_indices</code> and the previous name has been deprecated.</p>
+
+<p><code class="highlighter-rouge">GArrowField</code> supports functions to handle metadata.
+<code class="highlighter-rouge">GArrowSchema</code> supports <code class="highlighter-rouge">garrow_schema_has_metadata()</code> function.</p>
+
+<p><code class="highlighter-rouge">GArrowArrayBuilder</code> supports to add single null, multiple nulls, single empty value, and multiple empty values.
+<code class="highlighter-rouge">GArrowFixedSizedBinaryArrayBuilder</code> is newly supported.</p>
+
+<p>256-bit decimal and extension types are newly supported.
+Filesystem module supports Mock, HDFS, S3 file systems.
+Dataset module supports CSV, IPC, and Parquet file formats.</p>
+
+<h2 id="rust-notes">Rust notes</h2>
+<h3 id="core-arrow-crate">Core Arrow Crate</h3>
+
+<p>The development of the arrow crate was focused on four main aspects:</p>
+
+<ul>
+ <li>Make the crate usable in stable Rust</li>
+ <li>Bug fixing and removal of <code class="highlighter-rouge">unsafe</code> code</li>
+ <li>Extend functionality to keep up with the specification</li>
+ <li>Increase performance of existing kernels</li>
+</ul>
+
+<h4 id="stable-rust">Stable Rust</h4>
+
+<p>Possibly the biggest news for this release is that all project crates, including arrow, parquet, and datafusion now
+build with stable Rust by default. Nightly / unstable Rust is still required when enabling the SIMD feature.</p>
+
+<h4 id="parquet-arrow-writer">Parquet Arrow writer</h4>
+
+<p>The Parquet Writer for Arrow arrays is now available, allowing the Rust programs to easily read and write Parquet
+files and making it easier to integrate with the overall Arrow ecosystem. The reader and writer include both basic
+and nested type support (List, Dictionary, Struct)</p>
+
+<h4 id="first-class-arrow-flight-ipc-support">First Class Arrow Flight IPC Support</h4>
+
+<p>This release the Arrow Flight IPC implementation in Rust became fully-featured enough to participate in the regular
+cross-language integration tests, thus ensuring Rust applications written using Arrow can interoperate with the rest
+of the ecosystem</p>
+
+<h4 id="performance">Performance</h4>
+
+<p>There have been numerous performance improvements in this release across the board. This includes both kernel
+operations, such as <code class="highlighter-rouge">take</code>, <code class="highlighter-rouge">filter</code>, and <code class="highlighter-rouge">cast</code>, as well as more fundamental parts such as bitwise comparison
+and reading and writing to CSV.</p>
+
+<h4 id="increased-data-type-support">Increased Data Type Support</h4>
+
+<p>New DataTypes:</p>
+<ul>
+ <li>Decimal data type for fixed-width decimal values</li>
+</ul>
+
+<p>Improved operation support for nested structures Dictionary, and Lists (filter, take, etc)</p>
+
+<h4 id="other-improvements">Other improvements:</h4>
+
+<ul>
+ <li>Added support for Date and time on FFI</li>
+ <li>Added support for Binary type on FFI</li>
+ <li>Added support for i64 sized arrays to “take” kernel</li>
+ <li>Support for the i128 Decimal Type</li>
+ <li>Added support to cast string to date</li>
+ <li>Added API to create arrays out of existing arrays (e.g. for join, merge-sort, concatenate)</li>
+ <li>The simd feature is now also available on aarch64</li>
+</ul>
+
+<h4 id="api-changes">API Changes</h4>
+
+<ul>
+ <li>BooleanArray is no longer a PrimitiveArray</li>
+ <li>ArrowNativeType no longer includes bool since arrows boolean type is represented using bitpacking</li>
+ <li>Several Buffer methods are now infallible instead of returning a Result</li>
+ <li>DataType::List now contains a Field to track metadata about the contained elements</li>
+ <li>PrimitiveArray::raw_values, values_slice and values methods got replaced by a values method returning a slice</li>
+ <li>Buffer::data and raw_data were renamed to as_slice and as_ptr</li>
+ <li>MutableBuffer::data_mut and freeze were renamed to as_slice_mut and into to be more consistent with the stdlib naming conventions</li>
+ <li>The generic type parameter for BufferBuilder was changed from ArrowPrimitiveType to ArrowNativeType</li>
+</ul>
+
+<h3 id="datafusion">DataFusion</h3>
+
+<h4 id="sql">SQL</h4>
+
+<p>In this release, we clarified that DataFusion will standardize on the PostgreSQL SQL dialect.</p>
+
+<p>New SQL support:</p>
+<ul>
+ <li>JOIN, LEFT JOIN, RIGHT JOIN</li>
+ <li>COUNT DISTINCT</li>
+ <li>CASE WHEN</li>
+ <li>USING</li>
+ <li>BETWEEN</li>
+ <li>IS IN</li>
+ <li>Nested SELECT statements</li>
+ <li>Nested expressions in aggregations</li>
+ <li>LOWER(), UPPER(), TRIM()</li>
+ <li>NULLIF()</li>
+ <li>SHA224(), SHA256(), SHA384(), SHA512()</li>
+ <li>DATE_TRUNC()</li>
+</ul>
+
+<h4 id="performance-1">Performance</h4>
+
+<p>There have been numerous performance improvements in this release:</p>
+<ul>
+ <li>Optimizations for JOINs such as using vectorized hashing.</li>
+ <li>We started with adding statistics and cost-based optimizations. We choose the smaller side of a join as the build
+side if possible.</li>
+ <li>Improved parallelism when reading partitioned Parquet data sources</li>
+ <li>Concurrent writes of CSV and Parquet partitions to file</li>
+</ul>
+
+<h3 id="parquet-crate">Parquet Crate</h3>
+
+<p>The Parquet has the following improvements:</p>
+
+<ul>
+ <li>Nested reading</li>
+ <li>Support to write booleans</li>
+ <li>Add support to write temporal types</li>
+</ul>
+
+<h3 id="roadmap-for-400">Roadmap for 4.0.0</h3>
+
+<p>We have also started building up a shared community roadmap for 4.0: <a href="https://docs.google.com/document/d/1qspsOM_dknOxJKdGvKbC1aoVoO0M3i6x1CIo58mmN2Y/edit#heading=h.kstb571j5g5j">Apache Arrow: Crowd Sourced Rust Roadmap for
+Arrow 4.0, January 2021</a>.</p>
+
+
+ </main>
+ </div>
+
+ <hr/>
+<footer class="footer">
+ <div class="row">
+ <div class="col-md-9">
+ <p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
+ <p>© 2016-2021 The Apache Software Foundation</p>
+ </div>
+ <div class="col-md-3">
+ <a class="d-sm-none d-md-inline pr-2" href="https://www.apache.org/events/current-event.html">
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ </div>
+ <script src="/assets/main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js" type="text/javascript" integrity="sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM=" crossorigin="anonymous"></script>
+</footer>
+
+ </div>
+</body>
+</html>
diff --git a/blog/index.html b/blog/index.html
index 75aa281..a9003db 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -216,6 +216,21 @@
<p>
<h3>
+ <a href="/blog/2021/01/25/3.0.0-release/">Apache Arrow 3.0.0 Release</a>
+ </h3>
+
+ <p>
+ <span class="blog-list-date">
+ 25 January 2021
+ </span>
+ </p>
+ The Apache Arrow team is pleased to announce the 3.0.0 release. This covers over 3 months of development work and includes 666 resolved issues from 106 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The release notes below are not exhaustive and...
+ </p>
+
+
+
+ <p>
+ <h3>
<a href="/blog/2020/10/27/rust-2.0.0-release/">Apache Arrow 2.0.0 Rust Highlights</a>
</h3>
diff --git a/docs/c_glib/index.html b/docs/c_glib/index.html
new file mode 100644
index 0000000..d1a4b17
--- /dev/null
+++ b/docs/c_glib/index.html
@@ -0,0 +1,247 @@
+<!DOCTYPE html>
+<html lang="en-US">
+ <head>
+ <meta charset="UTF-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above meta tags *must* come first in the head; any other head content must come *after* these tags -->
+
+ <title>Apache Arrow GLib (C) | Apache Arrow</title>
+
+
+ <!-- Begin Jekyll SEO tag v2.7.1 -->
+<meta name="generator" content="Jekyll v3.8.4" />
+<meta property="og:title" content="Apache Arrow GLib (C)" />
+<meta property="og:locale" content="en_US" />
+<meta name="description" content="Apache Arrow GLib (C) Apache Arrow GLib is a wrapper library for Apache Arrow C++. Apache Arrow GLib provides C API. Apache Arrow GLib supports GObject Introspection. It means that you can create language bindings at runtime or compile time automatically. API reference manuals Apache Arrow GLib Apache Parquet GLib Gandiva GLib Plasma GLib" />
+<meta property="og:description" content="Apache Arrow GLib (C) Apache Arrow GLib is a wrapper library for Apache Arrow C++. Apache Arrow GLib provides C API. Apache Arrow GLib supports GObject Introspection. It means that you can create language bindings at runtime or compile time automatically. API reference manuals Apache Arrow GLib Apache Parquet GLib Gandiva GLib Plasma GLib" />
+<link rel="canonical" href="https://arrow.apache.org/docs/c_glib/" />
+<meta property="og:url" content="https://arrow.apache.org/docs/c_glib/" />
+<meta property="og:site_name" content="Apache Arrow" />
+<meta property="og:image" content="https://arrow.apache.org/img/arrow.png" />
+<meta name="twitter:card" content="summary_large_image" />
+<meta property="twitter:image" content="https://arrow.apache.org/img/arrow.png" />
+<meta property="twitter:title" content="Apache Arrow GLib (C)" />
+<meta name="twitter:site" content="@ApacheArrow" />
+<script type="application/ld+json">
+{"publisher":{"@type":"Organization","logo":{"@type":"ImageObject","url":"https://arrow.apache.org/img/logo.png"}},"description":"Apache Arrow GLib (C) Apache Arrow GLib is a wrapper library for Apache Arrow C++. Apache Arrow GLib provides C API. Apache Arrow GLib supports GObject Introspection. It means that you can create language bindings at runtime or compile time automatically. API reference manuals Apache Arrow GLib Apache Parquet GLib Gandiva GLib Plasma GLib","url":"https://arrow [...]
+<!-- End Jekyll SEO tag -->
+
+
+ <!-- favicons -->
+ <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16.png" id="light1">
+ <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32.png" id="light2">
+ <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon.png" id="light3">
+ <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120.png" id="light4">
+ <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76.png" id="light5">
+ <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60.png" id="light6">
+ <!-- dark mode favicons -->
+ <link rel="icon" type="image/png" sizes="16x16" href="/img/favicon-16x16-dark.png" id="dark1">
+ <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon-32x32-dark.png" id="dark2">
+ <link rel="apple-touch-icon" type="image/png" sizes="180x180" href="/img/apple-touch-icon-dark.png" id="dark3">
+ <link rel="apple-touch-icon" type="image/png" sizes="120x120" href="/img/apple-touch-icon-120x120-dark.png" id="dark4">
+ <link rel="apple-touch-icon" type="image/png" sizes="76x76" href="/img/apple-touch-icon-76x76-dark.png" id="dark5">
+ <link rel="apple-touch-icon" type="image/png" sizes="60x60" href="/img/apple-touch-icon-60x60-dark.png" id="dark6">
+
+ <script>
+ // Switch to the dark-mode favicons if prefers-color-scheme: dark
+ function onUpdate() {
+ light1 = document.querySelector('link#light1');
+ light2 = document.querySelector('link#light2');
+ light3 = document.querySelector('link#light3');
+ light4 = document.querySelector('link#light4');
+ light5 = document.querySelector('link#light5');
+ light6 = document.querySelector('link#light6');
+
+ dark1 = document.querySelector('link#dark1');
+ dark2 = document.querySelector('link#dark2');
+ dark3 = document.querySelector('link#dark3');
+ dark4 = document.querySelector('link#dark4');
+ dark5 = document.querySelector('link#dark5');
+ dark6 = document.querySelector('link#dark6');
+
+ if (matcher.matches) {
+ light1.remove();
+ light2.remove();
+ light3.remove();
+ light4.remove();
+ light5.remove();
+ light6.remove();
+ document.head.append(dark1);
+ document.head.append(dark2);
+ document.head.append(dark3);
+ document.head.append(dark4);
+ document.head.append(dark5);
+ document.head.append(dark6);
+ } else {
+ dark1.remove();
+ dark2.remove();
+ dark3.remove();
+ dark4.remove();
+ dark5.remove();
+ dark6.remove();
+ document.head.append(light1);
+ document.head.append(light2);
+ document.head.append(light3);
+ document.head.append(light4);
+ document.head.append(light5);
+ document.head.append(light6);
+ }
+ }
+ matcher = window.matchMedia('(prefers-color-scheme: dark)');
+ matcher.addListener(onUpdate);
+ onUpdate();
+ </script>
+
+ <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic,900">
+
+ <link href="/css/main.css" rel="stylesheet">
+ <link href="/css/syntax.css" rel="stylesheet">
+ <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js" integrity="sha384-ZMP7rVo3mIykV+2+9J3UJ46jBk0WLaUAdn689aCwoqbBJiSnjAK/l8WvCWPIPm49" crossorigin="anonymous"></script>
+
+ <!-- Global Site Tag (gtag.js) - Google Analytics -->
+<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
+<script>
+ window.dataLayer = window.dataLayer || [];
+ function gtag(){dataLayer.push(arguments)};
+ gtag('js', new Date());
+
+ gtag('config', 'UA-107500873-1');
+</script>
+
+
+ </head>
+
+
+<body class="wrap">
+ <header>
+ <nav class="navbar navbar-expand-md navbar-dark bg-dark">
+
+ <a class="navbar-brand no-padding" href="/"><img src="/img/arrow-inverse-300px.png" height="40px"/></a>
+
+ <button class="navbar-toggler ml-auto" type="button" data-toggle="collapse" data-target="#arrow-navbar" aria-controls="arrow-navbar" aria-expanded="false" aria-label="Toggle navigation">
+ <span class="navbar-toggler-icon"></span>
+ </button>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <div class="collapse navbar-collapse justify-content-end" id="arrow-navbar">
+ <ul class="nav navbar-nav">
+ <li class="nav-item"><a class="nav-link" href="/overview/" role="button" aria-haspopup="true" aria-expanded="false">Overview</a></li>
+ <li class="nav-item"><a class="nav-link" href="/faq/" role="button" aria-haspopup="true" aria-expanded="false">FAQ</a></li>
+ <li class="nav-item"><a class="nav-link" href="/blog" role="button" aria-haspopup="true" aria-expanded="false">Blog</a></li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownGetArrow" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Get Arrow
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownGetArrow">
+ <a class="dropdown-item" href="/install/">Install</a>
+ <a class="dropdown-item" href="/release/">Releases</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow">Source Code</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownDocumentation" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Documentation
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownDocumentation">
+ <a class="dropdown-item" href="/docs">Project Docs</a>
+ <a class="dropdown-item" href="/docs/format/Columnar.html">Format</a>
+ <hr/>
+ <a class="dropdown-item" href="/docs/c_glib">C GLib</a>
+ <a class="dropdown-item" href="/docs/cpp">C++</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow/blob/master/csharp/README.md">C#</a>
+ <a class="dropdown-item" href="https://godoc.org/github.com/apache/arrow/go/arrow">Go</a>
+ <a class="dropdown-item" href="/docs/java">Java</a>
+ <a class="dropdown-item" href="/docs/js">JavaScript</a>
+ <a class="dropdown-item" href="https://arrow.juliadata.org/stable/">Julia</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow/blob/master/matlab/README.md">MATLAB</a>
+ <a class="dropdown-item" href="/docs/python">Python</a>
+ <a class="dropdown-item" href="/docs/r">R</a>
+ <a class="dropdown-item" href="https://github.com/apache/arrow/blob/master/ruby/README.md">Ruby</a>
+ <a class="dropdown-item" href="https://docs.rs/crate/arrow/">Rust</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="/community/">Communication</a>
+ <a class="dropdown-item" href="/docs/developers/contributing.html">Contributing</a>
+ <a class="dropdown-item" href="https://issues.apache.org/jira/browse/ARROW">Issue Tracker</a>
+ <a class="dropdown-item" href="/committers/">Governance</a>
+ <a class="dropdown-item" href="/use_cases/">Use Cases</a>
+ <a class="dropdown-item" href="/powered_by/">Powered By</a>
+ <a class="dropdown-item" href="/security/">Security</a>
+ <a class="dropdown-item" href="https://www.apache.org/foundation/policies/conduct.html">Code of Conduct</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownASF" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ ASF Links
+ </a>
+ <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownASF">
+ <a class="dropdown-item" href="http://www.apache.org/">ASF Website</a>
+ <a class="dropdown-item" href="http://www.apache.org/licenses/">License</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/sponsorship.html">Donate</a>
+ <a class="dropdown-item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
+ <a class="dropdown-item" href="http://www.apache.org/security/">Security</a>
+ </div>
+ </li>
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </header>
+
+ <div class="container p-4 pt-5">
+ <main role="main" class="pb-5">
+ <!--
+
+-->
+
+<h1 id="apache-arrow-glib-c">Apache Arrow GLib (C)</h1>
+
+<p>Apache Arrow GLib is a wrapper library for Apache Arrow C++. Apache Arrow GLib provides C API.</p>
+
+<p>Apache Arrow GLib supports <a href="https://wiki.gnome.org/action/show/Projects/GObjectIntrospection">GObject Introspection</a>. It means that you can create language bindings at runtime or compile time automatically.</p>
+
+<h2 id="api-reference-manuals">API reference manuals</h2>
+
+<ul>
+ <li><a href="arrow-glib/">Apache Arrow GLib</a></li>
+ <li><a href="parquet-glib/">Apache Parquet GLib</a></li>
+ <li><a href="gandiva-glib/">Gandiva GLib</a></li>
+ <li><a href="plasma-glib/">Plasma GLib</a></li>
+</ul>
+
+
+ </main>
+
+ <hr/>
+<footer class="footer">
+ <div class="row">
+ <div class="col-md-9">
+ <p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
+ <p>© 2016-2021 The Apache Software Foundation</p>
+ </div>
+ <div class="col-md-3">
+ <a class="d-sm-none d-md-inline pr-2" href="https://www.apache.org/events/current-event.html">
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ </div>
+ <script src="/assets/main-18cd3029557f73c1ee82e41113127b04f6fcd84c56d9db0cb9c40ebe26ef6e33.js" type="text/javascript" integrity="sha256-GM0wKVV/c8HuguQRExJ7BPb82ExW2dsMucQOvibvbjM=" crossorigin="anonymous"></script>
+</footer>
+
+ </div>
+</body>
+</html>
diff --git a/feed.xml b/feed.xml
index 6ed1256..3b767bb 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,4 +1,329 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.8.4">Jekyll</generator><link href="https://arrow.apache.org/feed.xml" rel="self" type="application/atom+xml" /><link href="https://arrow.apache.org/" rel="alternate" type="text/html" /><updated>2021-01-25T20:26:09-05:00</updated><id>https://arrow.apache.org/feed.xml</id><title type="html">Apache Arrow</title><subtitle>Apache Arrow is a cross-language developm [...]
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.8.4">Jekyll</generator><link href="https://arrow.apache.org/feed.xml" rel="self" type="application/atom+xml" /><link href="https://arrow.apache.org/" rel="alternate" type="text/html" /><updated>2021-02-03T14:35:33-05:00</updated><id>https://arrow.apache.org/feed.xml</id><title type="html">Apache Arrow</title><subtitle>Apache Arrow is a cross-language developm [...]
+
+-->
+
+<p>The Apache Arrow team is pleased to announce the 3.0.0 release. This covers
+over 3 months of development work and includes <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%203.0.0"><strong>666 resolved issues</strong></a>
+from <a href="/release/3.0.0.html#contributors"><strong>106 distinct contributors</strong></a>. See the Install Page to learn how to
+get the libraries for your platform.</p>
+
+<p>The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the <a href="/release/3.0.0.html">complete changelog</a>.</p>
+
+<h2 id="columnar-format-notes">Columnar Format Notes</h2>
+
+<p>The Decimal256 data type, which was already supported by the Arrow columnar
+format specification, is now implemented in C++ and Java (ARROW-9747).</p>
+
+<h2 id="arrow-flight-rpc-notes">Arrow Flight RPC notes</h2>
+
+<p>Authentication in C++/Java/Python has been overhauled, allowing more flexible authentication methods and use of standard headers.
+Support for cookies has also been added.
+The C++/Java implementations are now more permissive when parsing messages in order to interoperate better with other Flight implementations.</p>
+
+<p>A basic Flight implementation for C#/.NET has been added.
+See the <a href="https://arrow.apache.org/docs/status.html#flight-rpc">implementation status matrix</a> for details.</p>
+
+<h2 id="c-notes">C++ notes</h2>
+
+<p>The default memory pool can now be changed at runtime using the environment
+variable <code class="highlighter-rouge">ARROW_DEFAULT_MEMORY_POOL</code> (ARROW-11009). The environment variable
+is inspected at process startup. This is useful when trying to diagnose memory
+consumption issues with Arrow.</p>
+
+<p>STL-like iterators are now provided over concrete arrays. Those are useful for
+non-performance critical tasks, for example testing (ARROW-10776).</p>
+
+<p>It is now possible to concatenate dictionary arrays with unequal dictionaries.
+The dictionaries are unified when concatenating, for supported data types
+(ARROW-5336).</p>
+
+<p>Threads in a thread pool are now spawned lazily as needed for enqueued
+tasks, up to the configured capacity. They used to be spawned upfront on
+creation of the thread pool (ARROW-10038).</p>
+
+<h3 id="compute-layer">Compute layer</h3>
+
+<p>Comprehensive documentation for compute functions is now available:
+https://arrow.apache.org/docs/cpp/compute.html</p>
+
+<p>Compute functions for string processing have been added for:</p>
+
+<ul>
+ <li>splitting on whitespace (ASCII and Unicode flavors) and splitting on a
+pattern (ARROW-9991);</li>
+ <li>trimming characters (ARROW-9128).</li>
+</ul>
+
+<p>Behavior of the <code class="highlighter-rouge">index_in</code> and <code class="highlighter-rouge">is_in</code> compute functions with nulls has been
+changed for consistency (ARROW-10663).</p>
+
+<p>Multiple-column sort kernels are now available for tables and record batches
+(ARROW-8199, ARROW-10796, ARROW-10790).</p>
+
+<p>Performance of table filtering has been vastly improved (ARROW-10569).</p>
+
+<p>Scalar arguments are now accepted for more compute functions.</p>
+
+<p>Compute functions <code class="highlighter-rouge">quantile</code> (ARROW-10831) and <code class="highlighter-rouge">is_nan</code> (ARROW-11043) have been
+added for numeric data.</p>
+
+<p>Aggregation functions <code class="highlighter-rouge">any</code> (ARROW-1846) and <code class="highlighter-rouge">all</code> (ARROW-10301) have been
+added for boolean data.</p>
+
+<h3 id="dataset">Dataset</h3>
+
+<p>The <code class="highlighter-rouge">Expression</code> hierarchy has simplified to a wrapper around literals, field references,
+or calls to named functions. This enables usage of any compute function while filtering
+with no boilerplate.</p>
+
+<p>Parquet statistics are lazily parsed in <code class="highlighter-rouge">ParquetDatasetFactory</code> and
+<code class="highlighter-rouge">ParquetFileFragment</code> for shorter construction time.</p>
+
+<h3 id="csv">CSV</h3>
+
+<p>Conversion of string columns is now faster thanks to faster UTF-8 validation
+of small strings (ARROW-10313).</p>
+
+<p>Conversion of floating-point columns is now faster thanks to optimized
+string-to-double conversion routines (ARROW-10328).</p>
+
+<p>Parsing of ISO8601 timestamps is now more liberal: trailing zeros can
+be omitted in the fractional part (ARROW-10337).</p>
+
+<p>Fixed a bug where null detection could give the wrong results on some platforms
+(ARROW-11067).</p>
+
+<p>Added type inference for Date32 columns for values in the form <code class="highlighter-rouge">YYYY-MM-DD</code>
+(ARROW-11247).</p>
+
+<h3 id="feather">Feather</h3>
+
+<p>Fixed reading of compressed Feather files written with Arrow 0.17 (ARROW-11163).</p>
+
+<h3 id="filesystem-layer">Filesystem layer</h3>
+
+<p>S3 recursive tree walks now benefit from a parallel implementation, where reads
+of multiple child directories are now issued concurrently (ARROW-10788).</p>
+
+<p>Improved empty directory detection to be mindful of differences between Amazon
+and Minio S3 implementations (ARROW-10942).</p>
+
+<h3 id="flight-rpc">Flight RPC</h3>
+
+<p>IPv6 host addresses are now supported (ARROW-10475).</p>
+
+<h3 id="ipc">IPC</h3>
+
+<p>It is now possible to emit dictionary deltas where possible using the IPC
+stream writer. This is governed by a new variable in the <code class="highlighter-rouge">IpcWriteOptions</code> class
+(ARROW-6883).</p>
+
+<p>It is now possible to read wider tables, which used to fail due to reaching a
+limit during Flatbuffers verification (ARROW-10056).</p>
+
+<h3 id="parquet">Parquet</h3>
+
+<p>Fixed reading of LZ4-compressed Parquet columns emitted by the Java Parquet
+implementation (ARROW-11301).</p>
+
+<p>Fixed a bug where writing multiple batches of nullable nested strings to Parquet
+would not write any data in batches after the first one (ARROW-10493)</p>
+
+<p>The Decimal256 data type can be read from and written to Parquet (ARROW-10607).</p>
+
+<p>LargeString and LargeBinary data can now be written to Parquet (ARROW-10426).</p>
+
+<h2 id="c-notes-1">C# notes</h2>
+
+<p>The .NET package added initial support for Arrow Flight clients and servers. Support is enabled through two new NuGet packages <a href="https://www.nuget.org/packages/Apache.Arrow.Flight/">Apache.Arrow.Flight</a> (client) and <a href="https://www.nuget.org/packages/Apache.Arrow.Flight.AspNetCore/">Apache.Arrow.Flight.AspNetCore</a> (server).</p>
+
+<p>Also fixed an issue where ArrowStreamWriter wasn’t writing schema metadata to Arrow streams.</p>
+
+<h2 id="julia-notes">Julia notes</h2>
+
+<p>This is the first release to officially include
+<a href="https://github.com/apache/arrow/tree/master/julia/Arrow">an implementation</a>
+for the Julia language. The pure Julia implementation includes support
+for <a href="https://arrow.apache.org/docs/status.html">wide coverage of the format specification</a>.
+Additional details can be found in the
+<a href="https://julialang.org/blog/2021/01/arrow/">julialang.org blog post</a>.</p>
+
+<h2 id="python-notes">Python notes</h2>
+
+<p>Support for Python 3.9 was added (ARROW-10224), and support for Python 3.5
+was removed (ARROW-5679).</p>
+
+<p>Support for building manylinux1 packages has been removed (ARROW-11212).
+PyArrow continues to be available as manylinux2010 and manylinux2014 wheels.</p>
+
+<p>The minimal required version for NumPy is now 1.16.6. Note that when upgrading
+NumPy to 1.20, you also need to upgrade pyarrow to 3.0.0 to ensure compatibility,
+as this pyarrow release fixed a compatibility issue with NumPy 1.20 (ARROW-10833).</p>
+
+<p>Compute functions are now automatically exported from C++ to the <code class="highlighter-rouge">pyarrow.compute</code>
+module, and they have docstrings matching their C++ definition.</p>
+
+<p>An <code class="highlighter-rouge">iter_batches()</code> method is now available for reading a Parquet file iteratively
+(ARROW-7800).</p>
+
+<p>Alternate memory pools (such as mimalloc, jemalloc or the C malloc-based memory
+pool) are now available from Python (ARROW-11049).</p>
+
+<p>Fixed a potential deadlock when importing pandas from several threads (ARROW-10519).</p>
+
+<p>See the C++ notes above for additional details.</p>
+
+<h2 id="r-notes">R notes</h2>
+
+<p>This release contains new features for the Flight RPC wrapper, better
+support for saving R metadata (including <code class="highlighter-rouge">sf</code> spatial data) to Feather and
+Parquet files, several significant improvements to speed and memory management,
+and many other enhancements.</p>
+
+<p>For more on what’s in the 3.0.0 R package, see the <a href="/docs/r/news/">R changelog</a>.</p>
+
+<h2 id="ruby-and-c-glib-notes">Ruby and C GLib notes</h2>
+
+<h3 id="ruby">Ruby</h3>
+
+<p>In Ruby binding, 256-bit decimal support and <code class="highlighter-rouge">Arrow::FixedBinaryArrayBuilder</code> are added likewise C GLib below.</p>
+
+<h3 id="c-glib">C GLib</h3>
+
+<p>In the version 3.0.0 of C GLib consists of many new features.</p>
+
+<p>A chunked array, a record batch, and a table support <code class="highlighter-rouge">sort_indices</code> function as well as an array.
+These functions including array’s support to specify sorting option.
+<code class="highlighter-rouge">garrow_array_sort_to_indices</code> has been renamed to <code class="highlighter-rouge">garrow_array_sort_indices</code> and the previous name has been deprecated.</p>
+
+<p><code class="highlighter-rouge">GArrowField</code> supports functions to handle metadata.
+<code class="highlighter-rouge">GArrowSchema</code> supports <code class="highlighter-rouge">garrow_schema_has_metadata()</code> function.</p>
+
+<p><code class="highlighter-rouge">GArrowArrayBuilder</code> supports to add single null, multiple nulls, single empty value, and multiple empty values.
+<code class="highlighter-rouge">GArrowFixedSizedBinaryArrayBuilder</code> is newly supported.</p>
+
+<p>256-bit decimal and extension types are newly supported.
+Filesystem module supports Mock, HDFS, S3 file systems.
+Dataset module supports CSV, IPC, and Parquet file formats.</p>
+
+<h2 id="rust-notes">Rust notes</h2>
+<h3 id="core-arrow-crate">Core Arrow Crate</h3>
+
+<p>The development of the arrow crate was focused on four main aspects:</p>
+
+<ul>
+ <li>Make the crate usable in stable Rust</li>
+ <li>Bug fixing and removal of <code class="highlighter-rouge">unsafe</code> code</li>
+ <li>Extend functionality to keep up with the specification</li>
+ <li>Increase performance of existing kernels</li>
+</ul>
+
+<h4 id="stable-rust">Stable Rust</h4>
+
+<p>Possibly the biggest news for this release is that all project crates, including arrow, parquet, and datafusion now
+build with stable Rust by default. Nightly / unstable Rust is still required when enabling the SIMD feature.</p>
+
+<h4 id="parquet-arrow-writer">Parquet Arrow writer</h4>
+
+<p>The Parquet Writer for Arrow arrays is now available, allowing the Rust programs to easily read and write Parquet
+files and making it easier to integrate with the overall Arrow ecosystem. The reader and writer include both basic
+and nested type support (List, Dictionary, Struct)</p>
+
+<h4 id="first-class-arrow-flight-ipc-support">First Class Arrow Flight IPC Support</h4>
+
+<p>This release the Arrow Flight IPC implementation in Rust became fully-featured enough to participate in the regular
+cross-language integration tests, thus ensuring Rust applications written using Arrow can interoperate with the rest
+of the ecosystem</p>
+
+<h4 id="performance">Performance</h4>
+
+<p>There have been numerous performance improvements in this release across the board. This includes both kernel
+operations, such as <code class="highlighter-rouge">take</code>, <code class="highlighter-rouge">filter</code>, and <code class="highlighter-rouge">cast</code>, as well as more fundamental parts such as bitwise comparison
+and reading and writing to CSV.</p>
+
+<h4 id="increased-data-type-support">Increased Data Type Support</h4>
+
+<p>New DataTypes:</p>
+<ul>
+ <li>Decimal data type for fixed-width decimal values</li>
+</ul>
+
+<p>Improved operation support for nested structures Dictionary, and Lists (filter, take, etc)</p>
+
+<h4 id="other-improvements">Other improvements:</h4>
+
+<ul>
+ <li>Added support for Date and time on FFI</li>
+ <li>Added support for Binary type on FFI</li>
+ <li>Added support for i64 sized arrays to “take” kernel</li>
+ <li>Support for the i128 Decimal Type</li>
+ <li>Added support to cast string to date</li>
+ <li>Added API to create arrays out of existing arrays (e.g. for join, merge-sort, concatenate)</li>
+ <li>The simd feature is now also available on aarch64</li>
+</ul>
+
+<h4 id="api-changes">API Changes</h4>
+
+<ul>
+ <li>BooleanArray is no longer a PrimitiveArray</li>
+ <li>ArrowNativeType no longer includes bool since arrows boolean type is represented using bitpacking</li>
+ <li>Several Buffer methods are now infallible instead of returning a Result</li>
+ <li>DataType::List now contains a Field to track metadata about the contained elements</li>
+ <li>PrimitiveArray::raw_values, values_slice and values methods got replaced by a values method returning a slice</li>
+ <li>Buffer::data and raw_data were renamed to as_slice and as_ptr</li>
+ <li>MutableBuffer::data_mut and freeze were renamed to as_slice_mut and into to be more consistent with the stdlib naming conventions</li>
+ <li>The generic type parameter for BufferBuilder was changed from ArrowPrimitiveType to ArrowNativeType</li>
+</ul>
+
+<h3 id="datafusion">DataFusion</h3>
+
+<h4 id="sql">SQL</h4>
+
+<p>In this release, we clarified that DataFusion will standardize on the PostgreSQL SQL dialect.</p>
+
+<p>New SQL support:</p>
+<ul>
+ <li>JOIN, LEFT JOIN, RIGHT JOIN</li>
+ <li>COUNT DISTINCT</li>
+ <li>CASE WHEN</li>
+ <li>USING</li>
+ <li>BETWEEN</li>
+ <li>IS IN</li>
+ <li>Nested SELECT statements</li>
+ <li>Nested expressions in aggregations</li>
+ <li>LOWER(), UPPER(), TRIM()</li>
+ <li>NULLIF()</li>
+ <li>SHA224(), SHA256(), SHA384(), SHA512()</li>
+ <li>DATE_TRUNC()</li>
+</ul>
+
+<h4 id="performance-1">Performance</h4>
+
+<p>There have been numerous performance improvements in this release:</p>
+<ul>
+ <li>Optimizations for JOINs such as using vectorized hashing.</li>
+ <li>We started with adding statistics and cost-based optimizations. We choose the smaller side of a join as the build
+side if possible.</li>
+ <li>Improved parallelism when reading partitioned Parquet data sources</li>
+ <li>Concurrent writes of CSV and Parquet partitions to file</li>
+</ul>
+
+<h3 id="parquet-crate">Parquet Crate</h3>
+
+<p>The Parquet has the following improvements:</p>
+
+<ul>
+ <li>Nested reading</li>
+ <li>Support to write booleans</li>
+ <li>Add support to write temporal types</li>
+</ul>
+
+<h3 id="roadmap-for-400">Roadmap for 4.0.0</h3>
+
+<p>We have also started building up a shared community roadmap for 4.0: <a href="https://docs.google.com/document/d/1qspsOM_dknOxJKdGvKbC1aoVoO0M3i6x1CIo58mmN2Y/edit#heading=h.kstb571j5g5j">Apache Arrow: Crowd Sourced Rust Roadmap for
+Arrow 4.0, January 2021</a>.</p></content><author><name>pmc</name></author><category term="release" /><summary type="html">The Apache Arrow team is pleased to announce the 3.0.0 release. This covers over 3 months of development work and includes 666 resolved issues from 106 distinct contributors. See the Install Page to learn how to get the libraries for your platform. The release notes below are not exhaustive and only expose selected highlights of the release. Many other bu [...]
-->
@@ -1743,264 +2068,4 @@ Speed: 3129.63 MB/s
<p>実際に使っている例もあります。Dremioは<a href="https://github.com/dremio-hub/dremio-flight-connector">Arrow Flightベースの</a>コネクターを開発しました。このコネクターは<a href="https://www.dremio.com/is-time-to-replace-odbc-jdbc/">ODBCよりも20-50倍よい性能を発揮する</a>ことを示しました。ArrowのコントリビューターであるRyan MurrayはApache Sparkユーザー向けにFlight対応エンドポイントに接続する<a href="https://github.com/rymurr/flight-spark-source">データソース実装</a>を作りました。</p>
-<p>最後に今後の話をします。gRPCではない（あるいはTCPではない）データトランスポートをサポートできないか研究開発を進めるかもしれません。Flightの開発が進むとユーザーが使えるFlight対応サービスが増えていくでしょう。Flightは開発フレームワークなので、ユーザーが使うAPIは高レベルなAPIだけになるようにするつもりです。高レベルなAPIではFlightの詳細と特定のFlightアプリケーションに関連する詳細を隠します。</p></content><author><name>wesm</name></author><category term="application" /><category term="translation" /><summary type="html">この1.5年、Apache ArrowコミュニティーはFlightの設計と実装を進めてきました。Flightは高速なデータトランスポートを実現するための新しいクライアント・サーバー型のフレームワークです。Flightを使うとネットワーク�
��しに大きなデータセ [...]
-
--->
-
-<p>Over the last 18 months, the Apache Arrow community has been busy designing and
-implementing <strong>Flight</strong>, a new general-purpose client-server framework to
-simplify high performance transport of large datasets over network interfaces.</p>
-
-<p>Flight initially is focused on optimized transport of the Arrow columnar format
-(i.e. “Arrow record batches”) over <a href="https://grpc.io/">gRPC</a>, Google’s popular HTTP/2-based
-general-purpose RPC library and framework. While we have focused on integration
-with gRPC, as a development framework Flight is not intended to be exclusive to
-gRPC.</p>
-
-<p>One of the biggest features that sets apart Flight from other data transport
-frameworks is parallel transfers, allowing data to be streamed to or from a
-cluster of servers simultaneously. This enables developers to more easily
-create scalable data services that can serve a growing client base.</p>
-
-<p>In the 0.15.0 Apache Arrow release, we have ready-to-use Flight implementations
-in C++ (with Python bindings) and Java. These libraries are suitable for beta
-users who are comfortable with API or protocol changes while we continue to
-refine some low-level details in the Flight internals.</p>
-
-<h2 id="motivation">Motivation</h2>
-
-<p>Many people have experienced the pain associated with accessing large datasets
-over a network. There are many different transfer protocols and tools for
-reading datasets from remote data services, such as ODBC and JDBC. Over the
-last 10 years, file-based data warehousing in formats like CSV, Avro, and
-Parquet has become popular, but this also presents challenges as raw data must
-be transferred to local hosts before being deserialized.</p>
-
-<p>The work we have done since the beginning of Apache Arrow holds exciting
-promise for accelerating data transport in a number of ways. The <a href="https://github.com/apache/arrow/blob/master/docs/source/format/Columnar.rst">Arrow
-columnar format</a> has key features that can help us:</p>
-
-<ul>
- <li>It is an “on-the-wire” representation of tabular data that does not require
-deserialization on receipt</li>
- <li>Its natural mode is that of “streaming batches”, larger datasets are
-transported a batch of rows at a time (called “record batches” in Arrow
-parlance). In this post we will talk about “data streams”, these are
-sequences of Arrow record batches using the project’s binary protocol</li>
- <li>The format is language-independent and now has library support in 11
-languages and counting.</li>
-</ul>
-
-<p>Implementations of standard protocols like ODBC generally implement their own
-custom on-wire binary protocols that must be marshalled to and from each
-library’s public interface. The performance of ODBC or JDBC libraries varies
-greatly from case to case.</p>
-
-<p>Our design goal for Flight is to create a new protocol for data services that
-uses the Arrow columnar format as both the over-the-wire data representation as
-well as the public API presented to developers. In doing so, we reduce or
-remove the serialization costs associated with data transport and increase the
-overall efficiency of distributed data systems. Additionally, two systems that
-are already using Apache Arrow for other purposes can communicate data to each
-other with extreme efficiency.</p>
-
-<h2 id="flight-basics">Flight Basics</h2>
-
-<p>The Arrow Flight libraries provide a development framework for implementing a
-service that can send and receive data streams. A Flight server supports
-several basic kinds of requests:</p>
-
-<ul>
- <li><strong>Handshake</strong>: a simple request to determine whether the client is authorized
-and, in some cases, to establish an implementation-defined session token to
-use for future requests</li>
- <li><strong>ListFlights</strong>: return a list of available data streams</li>
- <li><strong>GetSchema</strong>: return the schema for a data stream</li>
- <li><strong>GetFlightInfo</strong>: return an “access plan” for a dataset of interest,
-possibly requiring consuming multiple data streams. This request can accept
-custom serialized commands containing, for example, your specific application
-parameters.</li>
- <li><strong>DoGet</strong>: send a data stream to a client</li>
- <li><strong>DoPut</strong>: receive a data stream from a client</li>
- <li><strong>DoAction</strong>: perform an implementation-specific action and return any
-results, i.e. a generalized function call</li>
- <li><strong>ListActions</strong>: return a list of available action types</li>
-</ul>
-
-<p>We take advantage of gRPC’s elegant “bidirectional” streaming support (built on
-top of <a href="https://grpc.io/docs/guides/concepts/">HTTP/2 streaming</a>) to allow clients and servers to send data and metadata
-to each other simultaneously while requests are being served.</p>
-
-<p>A simple Flight setup might consist of a single server to which clients connect
-and make <code class="highlighter-rouge">DoGet</code> requests.</p>
-
-<div align="center">
-<img src="/img/20191014_flight_simple.png" alt="Flight Simple Architecture" width="50%" class="img-responsive" />
-</div>
-
-<h2 id="optimizing-data-throughput-over-grpc">Optimizing Data Throughput over gRPC</h2>
-
-<p>While using a general-purpose messaging library like gRPC has numerous specific
-benefits beyond the obvious ones (taking advantage of all the engineering that
-Google has done on the problem), some work was needed to improve the
-performance of transporting large datasets. Many kinds of gRPC users only deal
-with relatively small messages, for example.</p>
-
-<p>The best-supported way to use gRPC is to define services in a <a href="https://github.com/protocolbuffers/protobuf">Protocol
-Buffers</a> (aka “Protobuf”) <code class="highlighter-rouge">.proto</code> file. A Protobuf plugin for gRPC
-generates gRPC service stubs that you can use to implement your
-applications. RPC commands and data messages are serialized using the <a href="https://developers.google.com/protocol-buffers/docs/encoding">Protobuf
-wire format</a>. Because we use “vanilla gRPC and Protocol Buffers”, gRPC
-clients that are ignorant of the Arrow columnar format can still interact with
-Flight services and handle the Arrow data opaquely.</p>
-
-<p>The main data-related Protobuf type in Flight is called <code class="highlighter-rouge">FlightData</code>. Reading
-and writing Protobuf messages in general is not free, so we implemented some
-low-level optimizations in gRPC in both C++ and Java to do the following:</p>
-
-<ul>
- <li>Generate the Protobuf wire format for <code class="highlighter-rouge">FlightData</code> including the Arrow record
-batch being sent without going through any intermediate memory copying or
-serialization steps.</li>
- <li>Reconstruct a Arrow record batch from the Protobuf representation of
-<code class="highlighter-rouge">FlightData</code> without any memory copying or deserialization. In fact, we
-intercept the encoded data payloads without allowing the Protocol Buffers
-library to touch them.</li>
-</ul>
-
-<p>In a sense we are “having our cake and eating it, too”. Flight implementations
-having these optimizations will have better performance, while naive gRPC
-clients can still talk to the Flight service and use a Protobuf library to
-deserialize <code class="highlighter-rouge">FlightData</code> (albeit with some performance penalty).</p>
-
-<p>As far as absolute speed, in our C++ data throughput benchmarks, we are seeing
-end-to-end TCP throughput in excess of 2-3GB/s on localhost without TLS
-enabled. This benchmark shows a transfer of ~12 gigabytes of data in about 4
-seconds:</p>
-
-<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./arrow-flight-benchmark <span class="nt">--records_per_stream</span> 100000000
-Bytes <span class="nb">read</span>: 12800000000
-Nanos: 3900466413
-Speed: 3129.63 MB/s
-</code></pre></div></div>
-
-<p>From this we can conclude that the machinery of Flight and gRPC adds relatively
-little overhead, and it suggests that many real-world applications of Flight
-will be bottlenecked on network bandwidth.</p>
-
-<h2 id="horizontal-scalability-parallel-and-partitioned-data-access">Horizontal Scalability: Parallel and Partitioned Data Access</h2>
-
-<p>Many distributed database-type systems make use of an architectural pattern
-where the results of client requests are routed through a “coordinator” and
-sent to the client. Aside from the obvious efficiency issues of transporting a
-dataset multiple times on its way to a client, it also presents a scalability
-problem for getting access to very large datasets.</p>
-
-<p>We wanted Flight to enable systems to create horizontally scalable data
-services without having to deal with such bottlenecks. A client request to a
-dataset using the <code class="highlighter-rouge">GetFlightInfo</code> RPC returns a list of <strong>endpoints</strong>, each of
-which contains a server location and a <strong>ticket</strong> to send that server in a
-<code class="highlighter-rouge">DoGet</code> request to obtain a part of the full dataset. To get access to the
-entire dataset, all of the endpoints must be consumed. While Flight streams are
-not necessarily ordered, we provide for application-defined metadata which can
-be used to serialize ordering information.</p>
-
-<p>This multiple-endpoint pattern has a number of benefits:</p>
-
-<ul>
- <li>Endpoints can be read by clients in parallel.</li>
- <li>The service that serves the <code class="highlighter-rouge">GetFlightInfo</code> “planning” request can delegate
-work to sibling services to take advantage of data locality or simply to help
-with load balancing.</li>
- <li>Nodes in a distributed cluster can take on different roles. For example, a
-subset of nodes might be responsible for planning queries while other nodes
-exclusively fulfill data stream (<code class="highlighter-rouge">DoGet</code> or <code class="highlighter-rouge">DoPut</code>) requests.</li>
-</ul>
-
-<p>Here is an example diagram of a multi-node architecture with split service
-roles:</p>
-
-<div align="center">
-<img src="/img/20191014_flight_complex.png" alt="Flight Complex Architecture" width="60%" class="img-responsive" />
-</div>
-
-<h2 id="actions-extending-flight-with-application-business-logic">Actions: Extending Flight with application business logic</h2>
-
-<p>While the <code class="highlighter-rouge">GetFlightInfo</code> request supports sending opaque serialized commands
-when requesting a dataset, a client may need to be able to ask a server to
-perform other kinds of operations. For example, a client may request for a
-particular dataset to be “pinned” in memory so that subsequent requests from
-other clients are served faster.</p>
-
-<p>A Flight service can thus optionally define “actions” which are carried out by
-the <code class="highlighter-rouge">DoAction</code> RPC. An action request contains the name of the action being
-performed and optional serialized data containing further needed
-information. The result of an action is a gRPC stream of opaque binary results.</p>
-
-<p>Some example actions:</p>
-
-<ul>
- <li>Metadata discovery, beyond the capabilities provided by the built-in
-<code class="highlighter-rouge">ListFlights</code> RPC</li>
- <li>Setting session-specific parameters and settings</li>
-</ul>
-
-<p>Note that it is not required for a server to implement any actions, and actions
-need not return results.</p>
-
-<h2 id="encryption-and-authentication">Encryption and Authentication</h2>
-
-<p>Flight supports encryption out of the box using gRPC’s built in TLS / OpenSSL
-capabilities.</p>
-
-<p>For authentication, there are extensible authentication handlers for the client
-and server that permit simple authentication schemes (like user and password)
-as well as more involved authentication such as Kerberos. The Flight protocol
-comes with a built-in <code class="highlighter-rouge">BasicAuth</code> so that user/password authentication can be
-implemented out of the box without custom development.</p>
-
-<h2 id="middleware-and-tracing">Middleware and Tracing</h2>
-
-<p>gRPC has the concept of “interceptors” which have allowed us to develop
-developer-defined “middleware” that can provide instrumentation of or telemetry
-for incoming and outgoing requests. One such framework for such instrumentation
-is <a href="https://opentracing.io/">OpenTracing</a>.</p>
-
-<p>Note that middleware functionality is one of the newest areas of the project
-and is only currently available in the project’s master branch.</p>
-
-<h2 id="grpc-but-not-only-grpc">gRPC, but not only gRPC</h2>
-
-<p>We specify server locations for <code class="highlighter-rouge">DoGet</code> requests using RFC 3986 compliant
-URIs. For example, TLS-secured gRPC may be specified like
-<code class="highlighter-rouge">grpc+tls://$HOST:$PORT</code>.</p>
-
-<p>While we think that using gRPC for the “command” layer of Flight servers makes
-sense, we may wish to support data transport layers other than TCP such as
-<a href="https://en.wikipedia.org/wiki/Remote_direct_memory_access">RDMA</a>. While some design and development work is required to make this
-possible, the idea is that gRPC could be used to coordinate get and put
-transfers which may be carried out on protocols other than TCP.</p>
-
-<h2 id="getting-started-and-whats-next">Getting Started and What’s Next</h2>
-
-<p>Documentation for Flight users is a work in progress, but the libraries
-themselves are mature enough for beta users that are tolerant of some minor API
-or protocol changes over the coming year.</p>
-
-<p>One of the easiest ways to experiment with Flight is using the Python API,
-since custom servers and clients can be defined entirely in Python without any
-compilation required. You can see an <a href="https://github.com/apache/arrow/tree/apache-arrow-0.15.0/python/examples/flight">example Flight client and server in
-Python</a> in the Arrow codebase.</p>
-
-<p>In real-world use, Dremio has developed an <a href="https://github.com/dremio-hub/dremio-flight-connector">Arrow Flight-based</a> connector
-which has been shown to <a href="https://www.dremio.com/is-time-to-replace-odbc-jdbc/">deliver 20-50x better performance over ODBC</a>. For
-Apache Spark users, Arrow contributor Ryan Murray has created a <a href="https://github.com/rymurr/flight-spark-source">data source
-implementation</a> to connect to Flight-enabled endpoints.</p>
-
-<p>As far as “what’s next” in Flight, support for non-gRPC (or non-TCP) data
-transport may be an interesting direction of research and development work. A
-lot of the Flight work from here will be creating user-facing Flight-enabled
-services. Since Flight is a development framework, we expect that user-facing
-APIs will utilize a layer of API veneer that hides many general Flight details
-and details related to a particular application of Flight in a custom data
-service.</p></content><author><name>wesm</name></author><category term="application" /><summary type="html">Over the last 18 months, the Apache Arrow community has been busy designing and implementing Flight, a new general-purpose client-server framework to simplify high performance transport of large datasets over network interfaces. Flight initially is focused on optimized transport of the Arrow columnar format (i.e. “Arrow record batches”) over gRPC, Google’s popular HTTP/2-base [...]
\ No newline at end of file
+<p>最後に今後の話をします。gRPCではない（あるいはTCPではない）データトランスポートをサポートできないか研究開発を進めるかもしれません。Flightの開発が進むとユーザーが使えるFlight対応サービスが増えていくでしょう。Flightは開発フレームワークなので、ユーザーが使うAPIは高レベルなAPIだけになるようにするつもりです。高レベルなAPIではFlightの詳細と特定のFlightアプリケーションに関連する詳細を隠します。</p></content><author><name>wesm</name></author><category term="application" /><category term="translation" /><summary type="html">この1.5年、Apache ArrowコミュニティーはFlightの設計と実装を進めてきました。Flightは高速なデータトランスポートを実現するための新しいクライアント・サーバー型のフレームワークです。Flightを使うとネットワーク�
��しに大きなデータセ [...]
\ No newline at end of file