You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@crail.apache.org by at...@apache.org on 2018/08/14 11:26:58 UTC
[3/3] incubator-crail-website git commit: Publishing from
679a82db8708973e929e31217e840ed6dcc51421
Publishing from 679a82db8708973e929e31217e840ed6dcc51421
Project: http://git-wip-us.apache.org/repos/asf/incubator-crail-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-crail-website/commit/59f881c8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-crail-website/tree/59f881c8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-crail-website/diff/59f881c8
Branch: refs/heads/asf-site
Commit: 59f881c8051943262adbb7105b630ef3dfcfab67
Parents: 015b827
Author: Animesh Trivedi <an...@gmail.com>
Authored: Tue Aug 14 13:26:15 2018 +0200
Committer: Animesh Trivedi <an...@gmail.com>
Committed: Tue Aug 14 13:26:15 2018 +0200
----------------------------------------------------------------------
content/blog/2018/08/sql-p1.html | 13 +-
content/documentation/docs.html | 97 -------
content/documentation/how-to-build.html | 351 -----------------------
content/feed.xml | 15 +-
content/img/blog/preview/sql-p1-summary.png | Bin 0 -> 58447 bytes
content/img/blog/sql-p1/albis-crail.svg | 2 +-
content/img/blog/sql-p1/outline.svg | 1 +
7 files changed, 19 insertions(+), 460 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/59f881c8/content/blog/2018/08/sql-p1.html
----------------------------------------------------------------------
diff --git a/content/blog/2018/08/sql-p1.html b/content/blog/2018/08/sql-p1.html
index 01f34a5..885bafd 100644
--- a/content/blog/2018/08/sql-p1.html
+++ b/content/blog/2018/08/sql-p1.html
@@ -111,13 +111,16 @@ This is the first blog post in a multi-part series where we will focus on relati
</ul>
<h3 id="overview">Overview</h3>
+
<p>In a typical cloud-based relational data processing setup, the input data is stored on an external data storage solution like HDFS or AWS S3. Data tables and their associated schema are converted into a storage-friendly format for optimal performance. Examples of some popular and familiar file formats are <a href="https://parquet.apache.org/">Apache Parquet</a>, <a href="https://orc.apache.org/">Apache ORC</a>, <a href="https://avro.apache.org/">Apache Avro</a>, <a href="https://en.wikipedia.org/wiki/JSON">JSON</a>, etc. More recently, <a href="https://arrow.apache.org/">Apache Arrow</a> has been introduced to standardize the in-memory columnar data representation between multiple frameworks. There is no one size fits all as all these formats have their own strengths, weaknesses, and features. In this blog, we are specifically interested in the performance of these formats on modern high-performance networking and storage devices.</p>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/outline.svg" width="550" /><figcaption>Figure 1: The benchmarking setup with HDFS and file formats on a 100 Gbps network with NVMe flash devices. All formats contains routines for compression, encoding, and value materialization with associated I/O buffer management and data copies routines.<p></p></figcaption></div></figure>
+
<p>To benchmark the performance of file formats, we wrote a set of micro-benchmarks which are available at <a href="https://github.com/zrlio/fileformat-benchmarks">https://github.com/zrlio/fileformat-benchmarks</a>. We cannot use typical SQL micro-benchmarks because every SQL engine has its own favorite file format, on which it performs the best. Hence, in order to ensure parity, we decoupled the performance of reading the input file format from the SQL query processing by writing simple table reading micro-benchmarks. Our benchmark reads in the store_sales table from the TPC-DS dataset (scale factor 100), and calculates a sum of values present in the table. The table contains 23 columns of integers, doubles, and longs.</p>
-<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/performance-all.svg" width="550" /><figcaption>Figure 1: Performance of JSON, Avro, Parquet, ORC, and Arrow on NVMe devices over a 100 Gbps network.<p></p></figcaption></div></figure>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/performance-all.svg" width="550" /><figcaption>Figure 2: Performance of JSON, Avro, Parquet, ORC, and Arrow on NVMe devices over a 100 Gbps network.<p></p></figcaption></div></figure>
-<p>We evaluate the performance of the benchmark on a 3 node HDFS cluster connected using 100 Gbps RoCE. One datanode in HDFS contains 4 NVMe devices with a collective aggregate bandwidth of 12.5 GB/sec (equals to 100 Gbps, hence, we have a balanced network and storage performance). Figure 1 shows our results where none of the file formats is able to deliver the full hardware performance for reading input files. One third of the performance is already lost in HDFS (maximum throughput 74.9 Gbps out of possible 100 Gbps). The rest of the performance is lost inside the file format implementation, which needs to deal with encoding, buffer and I/O management, compression, etc. The best performer is Apache Arrow which is designed for in-memory columnar datasets. The performance of these file formats are bounded by the performance of the CPU, which is 100% loaded during the experiment. For a detailed analysis of the file formats, please refer to our paper - <a href="https://www.usenix.org/c
onference/atc18/presentation/trivedi">Albis: High-Performance File Format for Big Data Systems (USENIX, ATC’18)</a>.</p>
+<p>We evaluate the performance of the benchmark on a 3 node HDFS cluster connected using 100 Gbps RoCE. One datanode in HDFS contains 4 NVMe devices with a collective aggregate bandwidth of 12.5 GB/sec (equals to 100 Gbps, hence, we have a balanced network and storage performance). Figure 2 shows our results where none of the file formats is able to deliver the full hardware performance for reading input files. One third of the performance is already lost in HDFS (maximum throughput 74.9 Gbps out of possible 100 Gbps). The rest of the performance is lost inside the file format implementation, which needs to deal with encoding, buffer and I/O management, compression, etc. The best performer is Apache Arrow which is designed for in-memory columnar datasets. The performance of these file formats are bounded by the performance of the CPU, which is 100% loaded during the experiment. For a detailed analysis of the file formats, please refer to our paper - <a href="https://www.usenix.org/c
onference/atc18/presentation/trivedi">Albis: High-Performance File Format for Big Data Systems (USENIX, ATC’18)</a>.</p>
<h3 id="albis-high-performance-file-format-for-big-data-systems">Albis: High-Performance File Format for Big Data Systems</h3>
@@ -129,7 +132,7 @@ This is the first blog post in a multi-part series where we will focus on relati
<li>Careful object materialization using a binary API: To optimize the runtime representation in managed runtimes like the JVM, only objects which are necessary for SQL processing are materialized. Otherwise, a 4 byte integer can be passed around as a byte array (using the binary API of Albis).</li>
</ul>
-<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/core-scalability.svg" width="550" /><figcaption>Figure 2: Core scalability of JSON, Avro, Parquet, ORC, Arrow, and Albis.<p></p></figcaption></div></figure>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/core-scalability.svg" width="550" /><figcaption>Figure 3: Core scalability of JSON, Avro, Parquet, ORC, Arrow, and Albis on HDFS/NVMe.<p></p></figcaption></div></figure>
<p>Using the Albis format, we revise our previous experiment where we read the input store_sales table from HDFS. In the figure above, we show the performance of Albis and other file formats with number of CPU cores involved. At the right hand of the x-axis, we have performance with all 16 cores engaged, hence, representing the peak possible performance. As evident, Albis delivers 59.9 Gbps out of 74.9 Gbps possible bandwidth with HDFS over NVMe. Albis performance is 1.9 - 21.4x better than other file formats. To give an impression where the performance is coming from, in the table below we show some micro-architectural features for Parquet, ORC, Arrow, and Albis. Our previously discussed design ideas in Albis result in a shorter code path (shown as less instructions required for each row), better cache performance (shows as lower cache misses per row), and clearly better performance (shown as nanoseconds required per row for processing). For a detailed evaluation of Albis please re
fer to our paper.</p>
@@ -168,9 +171,9 @@ This is the first blog post in a multi-part series where we will focus on relati
<h3 id="apache-crail-incubating-with-albis">Apache Crail (Incubating) with Albis</h3>
-<p>For our final experiment, we try to answer the question what it would take to deliver the full 100 Gbps bandwidth for Albis. Certainly, the first bottleneck is to improve the base storage layer performance. Here we use Apache Crail (Incubating) with its <a href="https://en.wikipedia.org/wiki/NVM_Express#NVMeOF">NVMeF</a> storage tier. This tier uses <a href="https://github.com/zrlio/jNVMf">jNVMf library</a> to implement NVMeF stack in Java. As we have shown in a previous blog <a href="http://crail.incubator.apache.org/blog/2017/08/crail-nvme-fabrics-v1.html">post</a> that Crail’s NVMeF tier can deliver performance (97.8 Gbps) very close to the hardware limits. Hence, Albis with Crail is a perfect setup to evaluate on high-performance NVMe and RDMA devices. Before we get there, let’s get some calculations right. The store_sales table in the TPC-DS dataset has a data density of 93.9% (out of 100 bytes, only 93.9 is data, others are null values). As we measure the goodput, the e
xpected performance of Albis on Crail is 93.9% of 97.8 Gbps, which calculates to 91.8 Gbps. In our experiments, Albis on Crail delivers 85.5 Gbps. Figure 2 shows more detailed results.</p>
+<p>For our final experiment, we try to answer the question what it would take to deliver the full 100 Gbps bandwidth for Albis. Certainly, the first bottleneck is to improve the base storage layer performance. Here we use Apache Crail (Incubating) with its <a href="https://en.wikipedia.org/wiki/NVM_Express#NVMeOF">NVMeF</a> storage tier. This tier uses <a href="https://github.com/zrlio/jNVMf">jNVMf library</a> to implement NVMeF stack in Java. As we have shown in a previous blog <a href="http://crail.incubator.apache.org/blog/2017/08/crail-nvme-fabrics-v1.html">post</a> that Crail’s NVMeF tier can deliver performance (97.8 Gbps) very close to the hardware limits. Hence, Albis with Crail is a perfect setup to evaluate on high-performance NVMe and RDMA devices. Before we get there, let’s get some calculations right. The store_sales table in the TPC-DS dataset has a data density of 93.9% (out of 100 bytes, only 93.9 is data, others are null values). As we measure the goodput, the e
xpected performance of Albis on Crail is 93.9% of 97.8 Gbps, which calculates to 91.8 Gbps. In our experiments, Albis on Crail delivers 85.5 Gbps. Figure 4 shows more detailed results.</p>
-<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/albis-crail.svg" width="550" /><figcaption>Figure 2: Performance of Albis on Crail.<p></p></figcaption></div></figure>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/albis-crail.svg" width="550" /><figcaption>Figure 4: Performance of Albis on Crail.<p></p></figcaption></div></figure>
<p>The left half of the figure shows the performance scalability of Albis on Crail in a setup with 1 core (8.9 Gbps) to 16 cores (85.5 Gbps). In comparison, the right half of the figure shows the performance of Crail on HDFS/NVMe at 59.9 Gbps, and on Crail/NVMe at 85.5 Gbps. The last bar shows the performance of Albis if the benchmark does not materialize Java object values. In this configuration, Albis on Crail delivers 91.3 Gbps, which is very close to the expected peak of 91.8 Gbps.</p>
http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/59f881c8/content/documentation/docs.html
----------------------------------------------------------------------
diff --git a/content/documentation/docs.html b/content/documentation/docs.html
deleted file mode 100644
index 43b351b..0000000
--- a/content/documentation/docs.html
+++ /dev/null
@@ -1,97 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <title>The Apache Crail (Incubating) Project: Presentations and Papers</title>
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <link href="http://crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet">
- <link href="http://crail.incubator.apache.org/css/group.css" rel="stylesheet">
- <link rel="alternate" type="application/atom+xml" title="Atom"
- href="http://crail.incubator.apache.org/blog/blog.xml">
-
- <meta property="og:image" content="http://crail.incubator.apache.org/img/blog/preview/docs-summary.png" />
- <meta property="og:image:secure_url" content="http://crail.incubator.apache.org/img/blog/preview/docs-summary.png" />
- </head>
-
- <body>
- <div class="container">
- <div class="header">
- <ul class="nav nav-pills pull-right">
-
-
-
- <li >
- <a href="http://crail.incubator.apache.org/">
- Home
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/overview/">
- Overview
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/download/">
- Downloads
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/blog/">
- Blog
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/community/">
- Community
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/documentation/">
- Documentation
- </a>
- </li>
-
- </ul>
- <a href="http://crail.incubator.apache.org/">
- <img src="http://crail.incubator.apache.org/img/crail_logo.png"
- srcset="http://crail.incubator.apache.org/img/crail_logo.png"
- alt="Crail" id="logo">
- </a>
- </div>
-
-
-
- <h2>Presentations and Papers</h2>
-
-
- <ul>
- <li>Dataworks Summit 2018, San Jose:</li>
- <li>Spark Summit 2018:</li>
- <li>Spark Summit 2017:</li>
-</ul>
-
-
- <br>
- <br>
- <div class="footer">
- <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/">incubation</a> at <a href="https://www.apache.org/">The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
- </p>
- </div>
-
- </div> <!-- /container -->
-
- <!-- Support retina images. -->
- <script type="text/javascript"
- src="http://crail.incubator.apache.org/js/srcset-polyfill.js"></script>
- </body>
-</html>
http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/59f881c8/content/documentation/how-to-build.html
----------------------------------------------------------------------
diff --git a/content/documentation/how-to-build.html b/content/documentation/how-to-build.html
deleted file mode 100644
index a7a2753..0000000
--- a/content/documentation/how-to-build.html
+++ /dev/null
@@ -1,351 +0,0 @@
-<!DOCTYPE html>
-<html>
- <head>
- <meta charset="utf-8">
- <title>The Apache Crail (Incubating) Project: Building instructions</title>
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <link href="http://crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet">
- <link href="http://crail.incubator.apache.org/css/group.css" rel="stylesheet">
- <link rel="alternate" type="application/atom+xml" title="Atom"
- href="http://crail.incubator.apache.org/blog/blog.xml">
-
- <meta property="og:image" content="http://crail.incubator.apache.org/img/blog/preview/how-to-build-summary.png" />
- <meta property="og:image:secure_url" content="http://crail.incubator.apache.org/img/blog/preview/how-to-build-summary.png" />
- </head>
-
- <body>
- <div class="container">
- <div class="header">
- <ul class="nav nav-pills pull-right">
-
-
-
- <li >
- <a href="http://crail.incubator.apache.org/">
- Home
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/overview/">
- Overview
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/download/">
- Downloads
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/blog/">
- Blog
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/community/">
- Community
- </a>
- </li>
-
-
- <li >
- <a href="http://crail.incubator.apache.org/documentation/">
- Documentation
- </a>
- </li>
-
- </ul>
- <a href="http://crail.incubator.apache.org/">
- <img src="http://crail.incubator.apache.org/img/crail_logo.png"
- srcset="http://crail.incubator.apache.org/img/crail_logo.png"
- alt="Crail" id="logo">
- </a>
- </div>
-
-
-
- <h2>Building instructions</h2>
-
-
- <h2 id="requirements">Requirements</h2>
-
-<ul>
- <li>Java 8 or higher</li>
- <li>RDMA-based network, e.g., Infiniband, iWARP, RoCE. There are two options to run Crail without RDMA networking hardware: (a) use SoftiWARP, (b) us the TCP/DRAM storage tier</li>
- <li>Libdisni.so, available as part of <a href="https://github.com/zrlio/disni">DiSNI</a></li>
-</ul>
-
-<h2 id="building">Building</h2>
-
-<p>To build Crail from source using <a href="http://maven.apache.org/">Apache Maven</a> execute the following steps:</p>
-
-<ol>
- <li>Obtain a copy of <a href="https://github.com/apache/incubator-crail">Crail</a> from Github</li>
- <li>Run: mvn -DskipTests install</li>
- <li>Copy tarball to the cluster and unpack it using tar xvfz crail-1.0-bin.tar.gz</li>
-</ol>
-
-<p>Note: later, when deploying Crail, make sure libdisni.so is part of your LD_LIBRARY_PATH. The easiest way to make it work is to copy libdisni.so into crail-1.0/lib</p>
-
-<h2 id="configuration">Configuration</h2>
-
-<p>To configure Crail use crail-site.conf.template as a basis and modify it to match your environment.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd crail-1.0/conf
-mv crail-site.conf.template crail-site.conf
-</code></pre></div></div>
-
-<p>There are a general file system properties and specific properties for the different storage tiers. A typical configuration for the general file system section may look as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.namenode.address crail://namenode:9060
-crail.storage.types org.apache.crail.storage.rdma.RdmaStorageTier
-crail.cachepath /dev/hugepages/cache
-crail.cachelimit 12884901888
-crail.blocksize 1048576
-crail.buffersize 1048576
-</code></pre></div></div>
-
-<p>In this configuration the namenode is configured to run using port 9060 on host ‘namenode’, which must be a valid host in the cluster. We further configure a single storage tier, in this case the RDMA-based DRAM tier. The cachepath property needs to point to a directory that is used by the file system to allocate memory for the client cache. Up to cachelimit size, all the memory that is used by Crail will be allocated via mmap from this location. Ideally, the directory specified in cachepath points to a hugetlbfs mountpoint. Aside from the general properties, each storage tier needs to be configured separately.</p>
-
-<h3 id="rdmadram-storage">RDMA/DRAM Storage</h3>
-
-<p>For the RDMA/DRAM tier we need to specify the interface that should be used by the storage nodes.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.rdma.interface eth0
-</code></pre></div></div>
-
-<p>The datapath property specifies a path from which the storage nodes will allocate blocks of memory via mmap. Again, that path best points to a hugetlbfs mountpoint.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.rdma.datapath /memory/data
-</code></pre></div></div>
-
-<p>You want to specify how much DRAM each datanode should donate into the file system pool using the <code class="highlighter-rouge">storagelimit</code> property. DRAM is allocated in chunks of <code class="highlighter-rouge">allocationsize</code>, which needs to be a multiple of <code class="highlighter-rouge">crail.blocksize</code>.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.rdma.allocationsize 1073741824
-crail.storage.rdma.storagelimit 75161927680
-</code></pre></div></div>
-
-<p>Crail supports optimized local operations via memcpy (instead of RDMA) in case a given file operation is backed by a local storage node. The indexpath specifies where Crail will store the necessary metadata that make these optimizations possible. Important: the indexpath must NOT point to a hugetlbfs mountpoint because index files will be updated which not possible in hugetlbfs.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.rdma.localmap true
-crail.storage.rdma.indexpath /index
-</code></pre></div></div>
-
-<h3 id="nvmfflash-storage">NVMf/Flash Storage</h3>
-
-<p>Crail is a multi-tiered storage system. Additinoal tiers can be enabled by adding them to the configuration as follows.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.types org.apache.crail.storage.rdma.RdmaStorageTier,org.apache.crail.storage.nvmf.NvmfStorageTier
-</code></pre></div></div>
-
-<p>For the NVMf storage tier we need to configure the server IP that is used when listening for new connections. We also need to configure the PCI address of the flash device we want to use, as well as the huge page mount point to be used for allocating memory.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.nvmf.bindip 10.40.0.XX
-crail.storage.nvmf.pcieaddr 0000:11:00.0
-crail.storage.nvmf.hugedir /dev/hugepages
-crail.storage.nvmf.servermempool 512
-crail.storage.nvmf.clientmempool 512
-</code></pre></div></div>
-
-<h2 id="deploying">Deploying</h2>
-
-<p>For all deployments, make sure you define CRAIL_HOME on each machine to point to the top level Crail directory.</p>
-
-<h3 id="starting-crail-manually">Starting Crail manually</h3>
-
-<p>The simplest way to run Crail is to start it manually on just a handful nodes. You will need to start the Crail namenode, plus at least one datanode. To start the namenode execute the following command on the host that is configured to be the namenode:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd crail-1.0/
-./bin/crail namenode
-</code></pre></div></div>
-
-<p>To start a datanode run the following command on a host in the cluster (ideally this is a different physical machine than the one running the namenode):</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail datanode
-</code></pre></div></div>
-
-<p>Now you should have a small deployment up with just one datanode. In this case the datanode is of type RDMA/DRAM, which is the default datnode. If you want to start a different storage tier you can do so by passing a specific datanode class as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail datanode -t org.apache.crail.storage.nvmf.NvmfStorageTier
-</code></pre></div></div>
-
-<p>This would start the shared storage datanode. Note that configuration in crail-site.conf needs to have the specific properties set of this type of datanode, in order for this to work.</p>
-
-<h3 id="larger-deployments">Larger deployments</h3>
-
-<p>To run larger deployments start Crail using</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/start-crail.sh
-</code></pre></div></div>
-
-<p>Similarly, Crail can be stopped by using</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/stop-crail.sh
-</code></pre></div></div>
-
-<p>For this to work include the list of machines to start datanodes in conf/slaves. You can start multiple datanode of different types on the same host as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>host02-ib
-host02-ib -t org.apache.crail.storage.nvmf.NvmfStorageTier
-host03-ib
-</code></pre></div></div>
-
-<p>In this example, we are configuring a Crail cluster with 2 physical hosts but 3 datanodes and two different storage tiers.</p>
-
-<h2 id="crail-shell">Crail Shell</h2>
-
-<p>Crail provides an contains an HDFS adaptor, thus, you can interact with Crail using the HDFS shell:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail fs
-</code></pre></div></div>
-
-<p>Crail, however, does not implement the full HDFS shell functionality. The basic commands to copy file to/from Crail, or to move and delete files, will work.</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail fs -mkdir /test
-./bin/crail fs -ls /
-./bin/crail fs -copyFromLocal <path-to-local-file> /test
-./bin/crail fs -cat /test/<file-name>
-</code></pre></div></div>
-
-<p>For the Crail shell to work properly, the HDFS configuration in crail-1.0/conf/core-site.xml needs to be configured accordingly:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><configuration>
- <property>
- <name>fs.crail.impl</name>
- <value>org.apache.crail.hdfs.CrailHadoopFileSystem</value>
- </property>
- <property>
- <name>fs.defaultFS</name>
- <value>crail://namenode:9060</value>
- </property>
- <property>
- <name>fs.AbstractFileSystem.crail.impl</name>
- <value>org.apache.crail.hdfs.CrailHDFS</value>
- </property>
- </configuration>
-</code></pre></div></div>
-
-<p>Note that the Crail HDFS interface currently cannot provide the full performance of Crail due to limitations of the HDFS API. In particular, the HDFS <code class="highlighter-rouge">FSDataOutputStream</code> API only support heap-based <code class="highlighter-rouge">byte[]</code> arrays which requires a data copy. Moreover, HDFS operations are synchronous preventing efficient pipelining of operations. Instead, applications that seek the best performance should use the Crail interface directly, as shown next.</p>
-
-<h2 id="programming-against-crail">Programming against Crail</h2>
-
-<p>The best way to program against Crail is to use Maven. Make sure you have the Crail dependency specified in your application pom.xml file:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependency>
- <groupId>org.apache.crail</groupId>
- <artifactId>crail-client</artifactId>
- <version>1.0</version>
-</dependency>
-</code></pre></div></div>
-
-<p>Then, create a Crail client as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CrailConfiguration conf = new CrailConfiguration();
-CrailStore store = CrailStore.newInstance(conf);
-</code></pre></div></div>
-
-<p>Make sure the crail-1.0/conf directory is part of the classpath.</p>
-
-<p>Crail supports different file types. The simplest way to create a file in Crail is as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CrailFile file = store.create(filename, CrailNodeType.DATAFILE, CrailStorageClass.DEFAULT, CrailLocationClass.DEFAULT).get().syncDir();
-</code></pre></div></div>
-
-<p>Aside from the actual filename, the ‘create()’ call takes as input the storage and location classes which are preferences for the storage tier and physical location that this file should be created in. Crail tries to satisfy these preferences later when the file is written. In the example we do not request any particular storage or location affinity.</p>
-
-<p>This ‘create()’ command is non-blocking, calling ‘get()’ on the returning future object awaits the completion of the call. At that time, the file has been created, but its directory entry may not be visible. Therefore, the file may not yet show up in a file enumeration of the given parent directory. Calling ‘syncDir()’ waits to for the directory entry to be completed. Both the ‘get()’ and the ‘syncDir()’ operation can be deffered to a later time at which they may become non-blocking operations.</p>
-
-<p>Once the file is created, a file stream can be obtained for writing:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CrailBufferedOutputStream outstream = file.getBufferedOutputStream(1024);
-</code></pre></div></div>
-
-<p>Here, we create a buffered stream so that we can pass heap byte arrays as well. We could also create a non-buffered stream using</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CrailOutputStream outstream = file.getDirectOutputStream(1024);
-</code></pre></div></div>
-
-<p>In both cases, we pass a write hint (1024 in the example) that indicates to Crail how much data we are intending to write. This allows Crail to optimize metadatanode lookups. Crail never prefetches data, but it may fetch the metadata of the very next operation concurrently with the current data operation if the write hint allows to do so.</p>
-
-<p>Once the stream has been obtained, there exist various ways to write a file. The code snippet below shows the use of the asynchronous interface:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CrailBuffer dataBuf = fs.allocateBuffer();
-Future<DataResult> future = outputStream.write(dataBuf);
-...
-future.get();
-</code></pre></div></div>
-
-<p>Reading files works very similar to writing. There exist various examples in org.apache.crail.tools.CrailBenchmark.</p>
-
-<h2 id="tcp-storage-tiers-and-rpc-binding">TCP Storage Tiers and RPC binding</h2>
-
-<p>Crail is designed for user-level networking and storage. It does, however, also provide plain TCP-based storage backends for storage and RPC and, thus, can be run easily on any machine without requiring spspecial hardware support. The TCP storage backend can be enabled as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.storage.types org.apache.crail.storage.tcp.TcpStorageTier
-</code></pre></div></div>
-
-<p>The TCP RPC binding can be enabled as follows:</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crail.namenode.rpctype org.apache.crail.namenode.rpc.tcp.TcpNameNode
-</code></pre></div></div>
-
-<h2 id="benchmarks">Benchmarks</h2>
-
-<p>Crail provides a set of benchmark tools to measure the performance. Type</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail iobench
-</code></pre></div></div>
-
-<p>to get an overview of the available benchmarks. For instance, to benchmark the sequential write performance, type</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail iobench -t write -s 1048576 -k 102400 -f /tmp.dat
-</code></pre></div></div>
-
-<p>This will create a file of size 100G, written sequentially in a sequence of 1MB operations.</p>
-
-<p>To read a file sequentially, type</p>
-
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/crail iobench -t read -s 1048576 -k 102400 -f /tmp.dat
-</code></pre></div></div>
-
-<p>This command issues 102400 read operations of 1MB each.</p>
-
-<p>The tool also contains benchmarks to read files randomly, or to measure the performance of opening files, etc.</p>
-
-<h2 id="applications">Applications</h2>
-
-<p>Crail is used by <a href="https://github.com/zrlio/crail-spark-io">Crail-Spark-IO</a>, a high-performance shuffle engine for Spark. <a href="https://github.com/zrlio/crail-terasort">Crail-Terasort</a> is a fast sorting benchmark for Spark based on Crail.</p>
-
-<h2 id="contributions">Contributions</h2>
-
-<p>PRs are always welcome. Please fork, and make necessary modifications
-you propose, and let us know.</p>
-
-<h2 id="contact">Contact</h2>
-
-<p>Please join the Crail developer mailing list for discussions and notifications. The list is at:</p>
-
-<p>dev@crail.incubator.apache.org.</p>
-
-
- <br>
- <br>
- <div class="footer">
- <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/">incubation</a> at <a href="https://www.apache.org/">The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
- </p>
- </div>
-
- </div> <!-- /container -->
-
- <!-- Support retina images. -->
- <script type="text/javascript"
- src="http://crail.incubator.apache.org/js/srcset-polyfill.js"></script>
- </body>
-</html>
http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/59f881c8/content/feed.xml
----------------------------------------------------------------------
diff --git a/content/feed.xml b/content/feed.xml
index 15db918..3f03ab5 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.7.0">Jekyll</generator><link href="http://crail.incubator.apache.org//feed.xml" rel="self" type="application/atom+xml" /><link href="http://crail.incubator.apache.org//" rel="alternate" type="text/html" /><updated>2018-08-09T17:20:11+02:00</updated><id>http://crail.incubator.apache.org//</id><title type="html">The Apache Crail (Incubating) Project</title><entry><title type="html">Sql P1 News</title><link href="http://crail.incubator.apache.org//blog/2018/08/sql-p1-news.html" rel="alternate" type="text/html" title="Sql P1 News" /><published>2018-08-09T00:00:00+02:00</published><updated>2018-08-09T00:00:00+02:00</updated><id>http://crail.incubator.apache.org//blog/2018/08/sql-p1-news</id><content type="html" xml:base="http://crail.incubator.apache.org//blog/2018/08/sql-p1-news.html"><p>A new blog <a href="http://crail.incubator.apache.org/blo
g/2018/08/sql-p1.html">post</a> discussing file formats performance is now online</p></content><author><name></name></author><category term="news" /><summary type="html">A new blog post discussing file formats performance is now online</summary></entry><entry><title type="html">SQL Performance: Part 1 - Input File Formats</title><link href="http://crail.incubator.apache.org//blog/2018/08/sql-p1.html" rel="alternate" type="text/html" title="SQL Performance: Part 1 - Input File Formats" /><published>2018-08-08T00:00:00+02:00</published><updated>2018-08-08T00:00:00+02:00</updated><id>http://crail.incubator.apache.org//blog/2018/08/sql-p1</id><content type="html" xml:base="http://crail.incubator.apache.org//blog/2018/08/sql-p1.html"><div style="text-align: justify">
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.7.0">Jekyll</generator><link href="http://crail.incubator.apache.org//feed.xml" rel="self" type="application/atom+xml" /><link href="http://crail.incubator.apache.org//" rel="alternate" type="text/html" /><updated>2018-08-14T13:26:12+02:00</updated><id>http://crail.incubator.apache.org//</id><title type="html">The Apache Crail (Incubating) Project</title><entry><title type="html">Sql P1 News</title><link href="http://crail.incubator.apache.org//blog/2018/08/sql-p1-news.html" rel="alternate" type="text/html" title="Sql P1 News" /><published>2018-08-09T00:00:00+02:00</published><updated>2018-08-09T00:00:00+02:00</updated><id>http://crail.incubator.apache.org//blog/2018/08/sql-p1-news</id><content type="html" xml:base="http://crail.incubator.apache.org//blog/2018/08/sql-p1-news.html"><p>A new blog <a href="http://crail.incubator.apache.org/blo
g/2018/08/sql-p1.html">post</a> discussing file formats performance is now online</p></content><author><name></name></author><category term="news" /><summary type="html">A new blog post discussing file formats performance is now online</summary></entry><entry><title type="html">SQL Performance: Part 1 - Input File Formats</title><link href="http://crail.incubator.apache.org//blog/2018/08/sql-p1.html" rel="alternate" type="text/html" title="SQL Performance: Part 1 - Input File Formats" /><published>2018-08-08T00:00:00+02:00</published><updated>2018-08-08T00:00:00+02:00</updated><id>http://crail.incubator.apache.org//blog/2018/08/sql-p1</id><content type="html" xml:base="http://crail.incubator.apache.org//blog/2018/08/sql-p1.html"><div style="text-align: justify">
<p>
This is the first blog post in a multi-part series where we will focus on relational data processing performance (e.g., SQL) in presence of high-performance network and storage devices - the kind of devices that Crail targets. Relational data processing is one of the most popular and versatile workloads people run in the cloud. The general idea is that data is stored in tables with a schema, and is processed using a domain specific language like SQL. Examples of some popular systems that support such relational data analytics in the cloud are <a href="https://spark.apache.org/sql/">Apache Spark/SQL</a>, <a href="https://hive.apache.org/">Apache Hive</a>, <a href="https://impala.apache.org/">Apache Impala</a>, etc. In this post, we discuss the important first step in relational data processing, which is the reading of input data tables.
</p>
@@ -32,13 +32,16 @@ This is the first blog post in a multi-part series where we will focus on relati
</ul>
<h3 id="overview">Overview</h3>
+
<p>In a typical cloud-based relational data processing setup, the input data is stored on an external data storage solution like HDFS or AWS S3. Data tables and their associated schema are converted into a storage-friendly format for optimal performance. Examples of some popular and familiar file formats are <a href="https://parquet.apache.org/">Apache Parquet</a>, <a href="https://orc.apache.org/">Apache ORC</a>, <a href="https://avro.apache.org/">Apache Avro</a>, <a href="https://en.wikipedia.org/wiki/JSON">JSON</a>, etc. More recently, <a href="https://arrow.apache.org/">Apache Arrow</a> has been introduced to standardize the in-memory columnar data representation between multiple frameworks. There is no one size fits all as all these formats have their own strengths, weaknesses, and features. In this blog, we are specifically interested in the performance of these
formats on modern high-performance networking and storage devices.</p>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/outline.svg" width="550" /><figcaption>Figure 1: The benchmarking setup with HDFS and file formats on a 100 Gbps network with NVMe flash devices. All formats contains routines for compression, encoding, and value materialization with associated I/O buffer management and data copies routines.<p></p></figcaption></div></figure>
+
<p>To benchmark the performance of file formats, we wrote a set of micro-benchmarks which are available at <a href="https://github.com/zrlio/fileformat-benchmarks">https://github.com/zrlio/fileformat-benchmarks</a>. We cannot use typical SQL micro-benchmarks because every SQL engine has its own favorite file format, on which it performs the best. Hence, in order to ensure parity, we decoupled the performance of reading the input file format from the SQL query processing by writing simple table reading micro-benchmarks. Our benchmark reads in the store_sales table from the TPC-DS dataset (scale factor 100), and calculates a sum of values present in the table. The table contains 23 columns of integers, doubles, and longs.</p>
-<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/performance-all.svg" width="550" /><figcaption>Figure 1: Performance of JSON, Avro, Parquet, ORC, and Arrow on NVMe devices over a 100 Gbps network.<p></p></figcaption></div></figure>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/performance-all.svg" width="550" /><figcaption>Figure 2: Performance of JSON, Avro, Parquet, ORC, and Arrow on NVMe devices over a 100 Gbps network.<p></p></figcaption></div></figure>
-<p>We evaluate the performance of the benchmark on a 3 node HDFS cluster connected using 100 Gbps RoCE. One datanode in HDFS contains 4 NVMe devices with a collective aggregate bandwidth of 12.5 GB/sec (equals to 100 Gbps, hence, we have a balanced network and storage performance). Figure 1 shows our results where none of the file formats is able to deliver the full hardware performance for reading input files. One third of the performance is already lost in HDFS (maximum throughput 74.9 Gbps out of possible 100 Gbps). The rest of the performance is lost inside the file format implementation, which needs to deal with encoding, buffer and I/O management, compression, etc. The best performer is Apache Arrow which is designed for in-memory columnar datasets. The performance of these file formats are bounded by the performance of the CPU, which is 100% loaded during the experiment. For a detailed analysis of the file formats, please refer to our paper - <a href="https://ww
w.usenix.org/conference/atc18/presentation/trivedi">Albis: High-Performance File Format for Big Data Systems (USENIX, ATC’18)</a>.</p>
+<p>We evaluate the performance of the benchmark on a 3 node HDFS cluster connected using 100 Gbps RoCE. One datanode in HDFS contains 4 NVMe devices with a collective aggregate bandwidth of 12.5 GB/sec (equals to 100 Gbps, hence, we have a balanced network and storage performance). Figure 2 shows our results where none of the file formats is able to deliver the full hardware performance for reading input files. One third of the performance is already lost in HDFS (maximum throughput 74.9 Gbps out of possible 100 Gbps). The rest of the performance is lost inside the file format implementation, which needs to deal with encoding, buffer and I/O management, compression, etc. The best performer is Apache Arrow which is designed for in-memory columnar datasets. The performance of these file formats are bounded by the performance of the CPU, which is 100% loaded during the experiment. For a detailed analysis of the file formats, please refer to our paper - <a href="https://ww
w.usenix.org/conference/atc18/presentation/trivedi">Albis: High-Performance File Format for Big Data Systems (USENIX, ATC’18)</a>.</p>
<h3 id="albis-high-performance-file-format-for-big-data-systems">Albis: High-Performance File Format for Big Data Systems</h3>
@@ -50,7 +53,7 @@ This is the first blog post in a multi-part series where we will focus on relati
<li>Careful object materialization using a binary API: To optimize the runtime representation in managed runtimes like the JVM, only objects which are necessary for SQL processing are materialized. Otherwise, a 4 byte integer can be passed around as a byte array (using the binary API of Albis).</li>
</ul>
-<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/core-scalability.svg" width="550" /><figcaption>Figure 2: Core scalability of JSON, Avro, Parquet, ORC, Arrow, and Albis.<p></p></figcaption></div></figure>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/core-scalability.svg" width="550" /><figcaption>Figure 3: Core scalability of JSON, Avro, Parquet, ORC, Arrow, and Albis on HDFS/NVMe.<p></p></figcaption></div></figure>
<p>Using the Albis format, we revise our previous experiment where we read the input store_sales table from HDFS. In the figure above, we show the performance of Albis and other file formats with number of CPU cores involved. At the right hand of the x-axis, we have performance with all 16 cores engaged, hence, representing the peak possible performance. As evident, Albis delivers 59.9 Gbps out of 74.9 Gbps possible bandwidth with HDFS over NVMe. Albis performance is 1.9 - 21.4x better than other file formats. To give an impression where the performance is coming from, in the table below we show some micro-architectural features for Parquet, ORC, Arrow, and Albis. Our previously discussed design ideas in Albis result in a shorter code path (shown as less instructions required for each row), better cache performance (shows as lower cache misses per row), and clearly better performance (shown as nanoseconds required per row for processing). For a detailed evaluation of Albis ple
ase refer to our paper.</p>
@@ -89,9 +92,9 @@ This is the first blog post in a multi-part series where we will focus on relati
<h3 id="apache-crail-incubating-with-albis">Apache Crail (Incubating) with Albis</h3>
-<p>For our final experiment, we try to answer the question what it would take to deliver the full 100 Gbps bandwidth for Albis. Certainly, the first bottleneck is to improve the base storage layer performance. Here we use Apache Crail (Incubating) with its <a href="https://en.wikipedia.org/wiki/NVM_Express#NVMeOF">NVMeF</a> storage tier. This tier uses <a href="https://github.com/zrlio/jNVMf">jNVMf library</a> to implement NVMeF stack in Java. As we have shown in a previous blog <a href="http://crail.incubator.apache.org/blog/2017/08/crail-nvme-fabrics-v1.html">post</a> that Crail’s NVMeF tier can deliver performance (97.8 Gbps) very close to the hardware limits. Hence, Albis with Crail is a perfect setup to evaluate on high-performance NVMe and RDMA devices. Before we get there, let’s get some calculations right. The store_sales table in the TPC-DS dataset has a data density of 93.9% (out of 100 bytes, only
93.9 is data, others are null values). As we measure the goodput, the expected performance of Albis on Crail is 93.9% of 97.8 Gbps, which calculates to 91.8 Gbps. In our experiments, Albis on Crail delivers 85.5 Gbps. Figure 2 shows more detailed results.</p>
+<p>For our final experiment, we try to answer the question what it would take to deliver the full 100 Gbps bandwidth for Albis. Certainly, the first bottleneck is to improve the base storage layer performance. Here we use Apache Crail (Incubating) with its <a href="https://en.wikipedia.org/wiki/NVM_Express#NVMeOF">NVMeF</a> storage tier. This tier uses <a href="https://github.com/zrlio/jNVMf">jNVMf library</a> to implement NVMeF stack in Java. As we have shown in a previous blog <a href="http://crail.incubator.apache.org/blog/2017/08/crail-nvme-fabrics-v1.html">post</a> that Crail’s NVMeF tier can deliver performance (97.8 Gbps) very close to the hardware limits. Hence, Albis with Crail is a perfect setup to evaluate on high-performance NVMe and RDMA devices. Before we get there, let’s get some calculations right. The store_sales table in the TPC-DS dataset has a data density of 93.9% (out of 100 bytes, only
93.9 is data, others are null values). As we measure the goodput, the expected performance of Albis on Crail is 93.9% of 97.8 Gbps, which calculates to 91.8 Gbps. In our experiments, Albis on Crail delivers 85.5 Gbps. Figure 4 shows more detailed results.</p>
-<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/albis-crail.svg" width="550" /><figcaption>Figure 2: Performance of Albis on Crail.<p></p></figcaption></div></figure>
+<figure><div style="text-align:center"><img src="http://crail.incubator.apache.org/img/blog/sql-p1/albis-crail.svg" width="550" /><figcaption>Figure 4: Performance of Albis on Crail.<p></p></figcaption></div></figure>
<p>The left half of the figure shows the performance scalability of Albis on Crail in a setup with 1 core (8.9 Gbps) to 16 cores (85.5 Gbps). In comparison, the right half of the figure shows the performance of Crail on HDFS/NVMe at 59.9 Gbps, and on Crail/NVMe at 85.5 Gbps. The last bar shows the performance of Albis if the benchmark does not materialize Java object values. In this configuration, Albis on Crail delivers 91.3 Gbps, which is very close to the expected peak of 91.8 Gbps.</p>
http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/59f881c8/content/img/blog/preview/sql-p1-summary.png
----------------------------------------------------------------------
diff --git a/content/img/blog/preview/sql-p1-summary.png b/content/img/blog/preview/sql-p1-summary.png
new file mode 100644
index 0000000..87d80fa
Binary files /dev/null and b/content/img/blog/preview/sql-p1-summary.png differ