You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2019/01/18 07:20:50 UTC
svn commit: r1851604 [2/2] - in /kylin/site: ./ blog/ blog/2019/01/17/
blog/2019/01/17/cisco-throughput-5x/ cn/download/ download/
images/blog/cisco_throughput_5x/
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1851604&r1=1851603&r2=1851604&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Fri Jan 18 07:20:49 2019
@@ -19,11 +19,283 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Thu, 17 Jan 2019 18:33:24 -0800</pubDate>
- <lastBuildDate>Thu, 17 Jan 2019 18:33:24 -0800</lastBuildDate>
+ <pubDate>Thu, 17 Jan 2019 23:11:23 -0800</pubDate>
+ <lastBuildDate>Thu, 17 Jan 2019 23:11:23 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
+ <title>How Cisco's Big Data Team Improved the High Concurrent Throughput of Apache Kylin by 5x</title>
+ <description><h2 id="background">Background</h2>
+
+<p>As part of the development group of Ciscoâs Big Data team, one of our responsibilities is to provide BI reports to our stakeholders. Stakeholders rely on the reporting system to check the usage of Ciscoâs business offerings. These reports are also used as a reference for billing, so they are critical to our stakeholders and the business overall.</p>
+
+<p>The raw data for these reports is sourced across multiple tables in our Oracle database. The monthly data volume for one table is in the billions, and if a customer wants to run a report for one year, at least one billion to two billion rows of data need to be aggregated or processed through other operations. Additionally, all results need to be provided in a short amount of time. In the course of our research, we discovered Apache Kylin, a distributed preprocessing engine for massive datasets based on pre-calculation, which enables you to query those massive datasets at sub-second latency.</p>
+
+<p>With the simulation test using our production data, we found that Kylin was ideal for our needs and was indeed capable of providing aggregated results on one billion rows of data in one second. However, we still needed to undergo additional tests for another use case. For one stakeholder, we provide 15 charts displayed on a single page. The BI system will send REST API requests to Kylin to query the data for each chart asynchronously. Based on the production data volume, if there are 20 stakeholders viewing the report on one node, they will trigger 15*20 = 300 requests. The high concurrent query performance of Kylin is what we needed to test here.</p>
+
+<h2 id="the-testing-stage">The Testing Stage</h2>
+
+<p><strong>Precondition</strong>: To reduce the impact from network cost, we deployed the testing tools in the same network environment with Kylin. Meanwhile, we turned off the query cache for Kylin to make sure that each request was executed on the bottom layer.</p>
+
+<p><strong>Testing Tools</strong>: Aside from our traditional testing tool, Apache Jmeter, we also used another open source tool: Gatlin (<a href="https://gatling.io/">https://gatling.io/</a>) to test the same case. We excluded the impact from the tools.</p>
+
+<p><strong>Testing Strategy</strong>: We simulated user requests of different sizes by increasing the number of concurrent threads, tracking the average response in 60 seconds, finding the bottleneck for Kylin query responses, and observing the maximum response time and success rate.</p>
+
+<p><strong>Testing Results</strong>:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th style="text-align: center">Thread</th>
+ <th style="text-align: center">Handled Queries (in 60 seconds)</th>
+ <th style="text-align: center">Handled Queries (per second)</th>
+ <th style="text-align: center">Mean Response Time (ms)</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: center">1</td>
+ <td style="text-align: center">773</td>
+ <td style="text-align: center">13</td>
+ <td style="text-align: center">77</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">15</td>
+ <td style="text-align: center">3245</td>
+ <td style="text-align: center">54</td>
+ <td style="text-align: center">279</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">25</td>
+ <td style="text-align: center">3844</td>
+ <td style="text-align: center">64</td>
+ <td style="text-align: center">390</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">50</td>
+ <td style="text-align: center">4912</td>
+ <td style="text-align: center">82</td>
+ <td style="text-align: center">612</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">75</td>
+ <td style="text-align: center">5405</td>
+ <td style="text-align: center">90</td>
+ <td style="text-align: center">841</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">100</td>
+ <td style="text-align: center">5436</td>
+ <td style="text-align: center">91</td>
+ <td style="text-align: center">1108</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">150</td>
+ <td style="text-align: center">5434</td>
+ <td style="text-align: center">91</td>
+ <td style="text-align: center">1688</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>Resulting in the line chart as follows:</p>
+
+<p><img src="/images/blog/cisco_throughput_5x/handled_queries_1.png" alt="" width="500px" height="300px" /></p>
+
+<p>â</p>
+
+<p><strong>Finding</strong>: When the number of concurrent threads reach 75, executed queries per second reach a peak of 90. The number does not become better even as we continue to increase the threads. 90 concurrent query responses in one second only allows 90/15 = 6 users to view a report at the same time. Even when we extend the Kylin query nodes to 3, query capability with 18 users per second is far behind our business demands.</p>
+
+<h2 id="root-cause-analysis">Root Cause Analysis</h2>
+
+<p>After reading and analyzing the query engine code of Kylin, we learned that Kylinâs query performs parallel filtering and calculation in HBaseâs region server by launching HBase Coprocessor. Based on this information, we checked the resource usage of the HBase cluster. The count of RPC Tasks processed on the region server did not increase linearly with the number of Kylin query requests when high concurrent queries occured. We concluded that there was a thread block on the Kylin side.</p>
+
+<p>We used Flame graph and JProfile to collect and analyze data from the Kylin query node and could not find the root cause. Then we tried to catch a thread snapshot of Kylin with Jstack. Analyzing the Jstack log, we discovered the root cause of the bottleneck causing this concurrent query issue. The example is as follows (Kylin version 2.5.0):</p>
+
+<p>One thread is locked at sun.misc.URLClassPath.getNextLoader. TID is 0x000000048007a180ï¼</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>"Query e9c44a2d-6226-ff3b-f984-ce8489107d79-3425" #3425 daemon prio=5 os_prio=0 tid=0x000000000472b000 nid=0x1433 waiting ]
+ java.lang.Thread.State: BLOCKED (on object monitor)
+ at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:469)
+ - locked &lt;0x000000048007a180&gt; (a sun.misc.URLClassPath)
+ at sun.misc.URLClassPath.findResource(URLClassPath.java:214)
+ at java.net.URLClassLoader$2.run(URLClassLoader.java:569)
+ at java.net.URLClassLoader$2.run(URLClassLoader.java:567)
+ at java.security.AccessController.doPrivileged(Native Method)
+ at java.net.URLClassLoader.findResource(URLClassLoader.java:566)
+ at java.lang.ClassLoader.getResource(ClassLoader.java:1096)
+ at java.lang.ClassLoader.getResource(ClassLoader.java:1091)
+ at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1666)
+ at org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:338)
+</code></pre>
+</div>
+
+<p>43 threads were waiting to lock &lt;0x000000048007a180&gt; at the same time:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>"Query f1f0bbec-a3f7-04b2-1ac6-fd3e03a0232d-4002" #4002 daemon prio=5 os_prio=0 tid=0x00007f27e71e7800 nid=0x1676 waiting ]
+ java.lang.Thread.State: BLOCKED (on object monitor)
+ at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:469)
+ - waiting to lock &lt;0x000000048007a180&gt; (a sun.misc.URLClassPath)
+ at sun.misc.URLClassPath.findResource(URLClassPath.java:214)
+ at java.net.URLClassLoader$2.run(URLClassLoader.java:569)
+ at java.net.URLClassLoader$2.run(URLClassLoader.java:567)
+ at java.security.AccessController.doPrivileged(Native Method)
+ at java.net.URLClassLoader.findResource(URLClassLoader.java:566)
+ at java.lang.ClassLoader.getResource(ClassLoader.java:1096)
+ at java.lang.ClassLoader.getResource(ClassLoader.java:1091)
+ at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1666)
+ at org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:338)
+</code></pre>
+</div>
+
+<p>We found that the closest code logic to Kylin was:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:338)
+</code></pre>
+</div>
+
+<p>Further analyzing the Kylin source code showed we were getting close to the resolution.</p>
+
+<h2 id="code-analysis">Code Analysis</h2>
+
+<p>When Kylin query engine builds a request to HBase Coprocessor, it will export Kylin properties (various properties used in Kylin) as Strings. This issue is caused by the relative code logic.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>function private static OrderedProperties buildSiteOrderedProps()
+</code></pre>
+</div>
+
+<ul>
+ <li>Each thread will getResouce to load âkylin-defaults.propertiesâ (the default properties file that users cannot modify).</li>
+</ul>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>// 1. load default configurations from classpath.
+// we have a kylin-defaults.properties in kylin/core-common/src/main/resources
+URL resource = Thread.currentThread().getContextClassLoader().getResource("kylin-defaults.properties");
+Preconditions.checkNotNull(resource);
+logger.info("Loading kylin-defaults.properties from {}", resource.getPath());
+OrderedProperties orderedProperties = new OrderedProperties();
+loadPropertiesFromInputStream(resource.openStream(), orderedProperties);
+</code></pre>
+</div>
+
+<ul>
+ <li>Loop 10 times to getResouce for âkylin-defaultsâ + (i) + â.propertiesâ. Thread LOCKED occurs here.</li>
+</ul>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>for (int i = 0; i &lt; 10; i++) {
+String fileName = "kylin-defaults" + + ".properties";
+ URL additionalResource = Thread.currentThread().getContextClassLoader().getResource(fileName);
+ if (additionalResource != null) {
+ logger.info("Loading {} from {} ", fileName, additionalResource.getPath());
+ loadPropertiesFromInputStream(additionalResource.openStream(), orderedProperties);
+ }
+</code></pre>
+</div>
+
+<p>Those logics were introduced in 2017/6/7, with JIRA ID KYLIN-2659 <em>Refactor KylinConfig so that all the default configurations are hidden in kylin-defaults.properties</em> reported by Hongbin Ma.</p>
+
+<h2 id="issue-fixing">Issue Fixing</h2>
+
+<p>For the first part of the logic, because kylin-defaults.properties is built in kylin-core-common-xxxx.jar, thereâs no need to getResource for it every time. We moved this logic to getInstanceFromEnv(). This logic gets called only once when service starts.</p>
+
+<p>We found one regression issue when fixing this bug. One class, CubeVisitService, is a Coprocessor. It will use KylinConfig as util class to generate KylinConfig object. Itâs dangerous to induce any logic to load properties. Due to this, there is no Kylin.properties file in Coprocessor.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>buildDefaultOrderedProperties();
+</code></pre>
+</div>
+
+<p>For the second part, this design should be future-proof and allow users to define 10 default properties (and override with each other), but after a year and a half, this logic seemed to never be used. However, to reduce risk, we kept this logic because it only gets called once during service start up which resulted in an insignificant waste of additional time.</p>
+
+<h2 id="performance-testing-after-bug-fixes">Performance Testing After Bug Fixes</h2>
+
+<p>Based on the same data volume and testing environment, results were as follows:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th style="text-align: center">Thread</th>
+ <th style="text-align: center">Handled Queries (in 60 seconds)</th>
+ <th style="text-align: center">Handled Queries (per second)</th>
+ <th style="text-align: center">Mean Response Time (ms)</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: center">1</td>
+ <td style="text-align: center">2451</td>
+ <td style="text-align: center">41</td>
+ <td style="text-align: center">12</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">15</td>
+ <td style="text-align: center">12422</td>
+ <td style="text-align: center">207</td>
+ <td style="text-align: center">37</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">25</td>
+ <td style="text-align: center">15600</td>
+ <td style="text-align: center">260</td>
+ <td style="text-align: center">56</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">50</td>
+ <td style="text-align: center">18481</td>
+ <td style="text-align: center">308</td>
+ <td style="text-align: center">129</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">75</td>
+ <td style="text-align: center">21055</td>
+ <td style="text-align: center">351</td>
+ <td style="text-align: center">136</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">100</td>
+ <td style="text-align: center">24036</td>
+ <td style="text-align: center">400</td>
+ <td style="text-align: center">251</td>
+ </tr>
+ <tr>
+ <td style="text-align: center">150</td>
+ <td style="text-align: center">28014</td>
+ <td style="text-align: center">467</td>
+ <td style="text-align: center">277</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>And the resulting line chart:</p>
+
+<p><img src="/images/blog/cisco_throughput_5x/handled_queries_2.png" alt="" width="500px" height="300px" /></p>
+
+<p>When the concurrent threads reached 150, Kylin processed 467 requests per second. The concurrent query capability increased by five times with linear growth . It could be concluded then that the bottleneck was eliminated . We didnât increase the concurrent threads due to the Kylin query engineâs settings for cluster load balancing which meant that increasing the concurrent connections on a single node increased the workload on the Tomcat server (Max connection is 150 in Kylin by default). The thread blocking issue disappeared after re-collecting and analyzing the Jstack log.</p>
+
+<p>After the fix, each Kylin node could now handle requests for 467/15 = 31 users, which meets our business requirement. Additionally, Kylinâs concurrent query capability can be further improved by several times once we enable query cache, so it is more than sufficient to fulfill our needs.</p>
+
+<h2 id="summary">Summary</h2>
+
+<p>Apache Kylin lets you query massive datasets at sub-second latency, thanks to the pre-calculation design of cubes, the optimization of Apache Calcite operator in queries, and also the introduction of âPrepared Statement Cacheâ to reduce the cost of Calcite SQL parses. Query performance optimization is not easy. We need to pay more attention to impacts on the Kylin query engine when new features are introduced or bugs are fixed, since even a minor code change could spell disaster. Issues like these in high concurrency scenarios can often be hard to reproduce and analyze.</p>
+
+<p>Lastly, query performance testing should not be limited to a single or small set of queries. High concurrecy performance testing should take place considering actual business requirements. For enterprise reporting systems, 3 seconds is the user tolerance limit for new page loading, which includes page rendering and network consumption. Ultimately, the backend data service should provide a response within 1 second. This is indeed a big challenge in a business scenario with big data sets. Fortunately, Kylin easily meets this requirement.</p>
+
+<p>This issue has already been submitted on JIRA as <a href="https://issues.apache.org/jira/browse/KYLIN-3672">KYLIN-3672</a>, and released in Kylin v2.5.2. Thanks to Shaofeng Shi of Kyligence Inc. for help.</p>
+
+<p>ã1ã<a href="https://issues.apache.org/jira/browse/KYLIN-3672">https://issues.apache.org/jira/browse/KYLIN-3672</a></p>
+
+<p><em>Author Zongwie Li as a Cisco engineer and a team member in the companyâs Big Data architecture team, currently responsible for OLAP platform construction and customer business reporting systems.</em></p>
+
+</description>
+ <pubDate>Thu, 17 Jan 2019 09:30:00 -0800</pubDate>
+ <link>http://kylin.apache.org/blog/2019/01/17/cisco-throughput-5x/</link>
+ <guid isPermaLink="true">http://kylin.apache.org/blog/2019/01/17/cisco-throughput-5x/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Introduce data source SDK</title>
<description><h2 id="data-source-sdk">Data source SDK</h2>
@@ -1165,47 +1437,6 @@ kylin.engine.spark.rdd-partition-cut-mb=
<category>blog</category>
-
- </item>
-
- <item>
- <title>A new measure for Percentile precalculation</title>
- <description><h2 id="introduction">Introduction</h2>
-
-<p>Since Apache Kylin 2.0, thereâs a new measure for percentile precalculation, which aims at (sub-)second latency for <strong>approximate</strong> percentile analytics SQL queries. The implementation is based on <a href="https://github.com/tdunning/t-digest">t-digest</a> library under Apachee 2.0 license, which provides a high-effecient data structure to save aggregation counters and algorithm to calculate approximate result of percentile.</p>
-
-<h3 id="percentile">Percentile</h3>
-<p><em>From <a href="https://en.wikipedia.org/wiki/Percentile">wikipedia</a></em>: A <strong>percentile</strong> (or a <strong>centile</strong>) is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. For example, the 20th percentile is the value (or score) below which 20% of the observations may be found.</p>
-
-<p>In Apache Kylin, we support the similar SQL sytanx like Apache Hive, with a aggregation function called <strong>percentile(&lt;Number Column&gt;, &lt;Double&gt;)</strong>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">seller_id</span><span class="p">,</span> <span class="n">percentile</span><span class="p">(</span><span class="n">price</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">5</span><span class="p">)</span>
-<span class="k">FROM</span> <span class="n">test_kylin_fact</span>
-<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">seller_id</span>
-</code></pre>
-</div>
-
-<h3 id="how-to-use">How to use</h3>
-<p>If you know little about <em>Cubes</em>, please go to <a href="http://kylin.apache.org/docs20/tutorial/kylin_sample.html">QuickStart</a> first to learn basic knowledge.</p>
-
-<p>Firstly, you need to add this column as measure in data model.</p>
-
-<p><img src="/images/blog/percentile_1.png" alt="" /></p>
-
-<p>Secondly, create a cube and add a PERCENTILE measure.</p>
-
-<p><img src="/images/blog/percentile_2.png" alt="" /></p>
-
-<p>Finally, build the cube and try some query.</p>
-
-<p><img src="/images/blog/percentile_3.png" alt="" /></p>
-</description>
- <pubDate>Sat, 01 Apr 2017 15:22:22 -0700</pubDate>
- <link>http://kylin.apache.org/blog/2017/04/01/percentile-measure/</link>
- <guid isPermaLink="true">http://kylin.apache.org/blog/2017/04/01/percentile-measure/</guid>
-
-
- <category>blog</category>
</item>
Added: kylin/site/images/blog/cisco_throughput_5x/handled_queries_1.png
URL: http://svn.apache.org/viewvc/kylin/site/images/blog/cisco_throughput_5x/handled_queries_1.png?rev=1851604&view=auto
==============================================================================
Binary file - no diff available.
Propchange: kylin/site/images/blog/cisco_throughput_5x/handled_queries_1.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: kylin/site/images/blog/cisco_throughput_5x/handled_queries_2.png
URL: http://svn.apache.org/viewvc/kylin/site/images/blog/cisco_throughput_5x/handled_queries_2.png?rev=1851604&view=auto
==============================================================================
Binary file - no diff available.
Propchange: kylin/site/images/blog/cisco_throughput_5x/handled_queries_2.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream