You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@phoenix.apache.org by mu...@apache.org on 2014/02/15 02:00:06 UTC
svn commit: r1568572 - in /incubator/phoenix/site/publish: ./ language/
Author: mujtaba
Date: Sat Feb 15 01:00:06 2014
New Revision: 1568572
URL: http://svn.apache.org/r1568572
Log:
Fix markdown syntax - https://issues.apache.org/jira/browse/PHOENIX-55
Modified:
incubator/phoenix/site/publish/Phoenix-in-15-minutes-or-less.html
incubator/phoenix/site/publish/building.html
incubator/phoenix/site/publish/download.html
incubator/phoenix/site/publish/language/datatypes.html
incubator/phoenix/site/publish/language/functions.html
incubator/phoenix/site/publish/language/index.html
incubator/phoenix/site/publish/mr_dataload.html
incubator/phoenix/site/publish/performance.html
incubator/phoenix/site/publish/pig_integration.html
incubator/phoenix/site/publish/secondary_indexing.html
Modified: incubator/phoenix/site/publish/Phoenix-in-15-minutes-or-less.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/Phoenix-in-15-minutes-or-less.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/Phoenix-in-15-minutes-or-less.html (original)
+++ incubator/phoenix/site/publish/Phoenix-in-15-minutes-or-less.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-14
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -115,24 +115,49 @@
<h1>Phoenix in 15 minutes or less</h1>
</div>
<p><i><b>What is this new <a href="index.html">Phoenix</a> thing Iâve been hearing about?</b></i><br /> Phoenix is an open source SQL skin for HBase. You use the standard JDBC APIs instead of the regular HBase client APIs to create tables, insert data, and query your HBase data.</p>
-<p><i><b>Doesnât putting an extra layer between my application and HBase just slow things down?</b></i><br /> Actually, no. Phoenix achieves as good or likely better <a href="performance.html">performance</a> than if you hand-coded it yourself (not to mention with a heck of a lot less code) by: * compiling your SQL queries to native HBase scans * determining the optimal start and stop for your scan key * orchestrating the parallel execution of your scans * bringing the computation to the data by * pushing the predicates in your where clause to a server-side filter * executing aggregate queries through server-side hooks (called co-processors)</p>
-<p>In addition to these items, weâve got some interesting enhancements in the works to further optimize performance: * secondary indexes to improve performance for queries on non row key columns * stats gathering to improve parallelization and guide choices between optimizations * skip scan filter to optimize IN, LIKE, and OR queries * optional salting of row keys to evenly distribute write load</p>
-<p><i><b>Ok, so itâs fast. But why SQL? Itâs so 1970s</b></i><br /> Well, thatâs kind of the point: give folks something with which theyâre already familiar. What better way to spur the adoption of HBase? On top of that, using JDBC and SQL: * Reduces the amount of code users need to write * Allows for performance optimizations transparent to the user * Opens the door for leveraging and integrating lots of existing tooling</p>
+<p><i><b>Doesnât putting an extra layer between my application and HBase just slow things down?</b></i><br /> Actually, no. Phoenix achieves as good or likely better <a href="performance.html">performance</a> than if you hand-coded it yourself (not to mention with a heck of a lot less code) by:</p>
+<ul>
+ <li>compiling your SQL queries to native HBase scans</li>
+ <li>determining the optimal start and stop for your scan key</li>
+ <li>orchestrating the parallel execution of your scans</li>
+ <li>bringing the computation to the data by</li>
+ <li>pushing the predicates in your where clause to a server-side filter</li>
+ <li>executing aggregate queries through server-side hooks (called co-processors)</li>
+</ul>
+<p>In addition to these items, weâve got some interesting enhancements in the works to further optimize performance:</p>
+<ul>
+ <li>secondary indexes to improve performance for queries on non row key columns</li>
+ <li>stats gathering to improve parallelization and guide choices between optimizations</li>
+ <li>skip scan filter to optimize IN, LIKE, and OR queries</li>
+ <li>optional salting of row keys to evenly distribute write load</li>
+</ul>
+<p><i><b>Ok, so itâs fast. But why SQL? Itâs so 1970s</b></i><br /> Well, thatâs kind of the point: give folks something with which theyâre already familiar. What better way to spur the adoption of HBase? On top of that, using JDBC and SQL:</p>
+<ul>
+ <li>Reduces the amount of code users need to write</li>
+ <li>Allows for performance optimizations transparent to the user</li>
+ <li>Opens the door for leveraging and integrating lots of existing tooling</li>
+</ul>
<p><i><b>But how can SQL support my favorite HBase technique of x,y,z</b></i><br /> Didnât make it to the last HBase Meetup did you? SQL is just a way of expressing <i><b>what you want to get</b></i> not <i><b>how you want to get it</b></i>. Check out my <a class="externalLink" href="http://files.meetup.com/1350427/IntelPhoenixHBaseMeetup.ppt">presentation</a> for various existing and to-be-done Phoenix features to support your favorite HBase trick. Have ideas of your own? Weâd love to hear about them: file an <a href="issues.html">issue</a> for us and/or join our <a href="mailing_list.html">mailing list</a>.</p>
<p><i><b>Blah, blah, blah - I just want to get started!</b></i><br /> Ok, great! Just follow our <a href="download.html#Installation">install instructions</a>: * <a href="download.html">download</a> and expand our installation tar * copy the phoenix jar into the HBase lib directory of every region server * restart the region servers * add the phoenix client jar to the classpath of your HBase client * download and <a href="download.html#SQL-Client">setup SQuirrel</a> as your SQL client so you can issue adhoc SQL against your HBase cluster</p>
-<p><i><b>I donât want to download and setup anything else!</b></i><br /> Ok, fair enough - you can create your own SQL scripts and execute them using our command line tool instead. Letâs walk through an example now. In the bin directory of your install location: * Create us_population.sql file </p>
-<div>
- <pre><tt>CREATE TABLE IF NOT EXISTS us_population ( state CHAR(2) NOT NULL, city VARCHAR NOT NULL, population BIGINT CONSTRAINT my_pk PRIMARY KEY (state, city));</tt></pre>
-</div> * Create us_population.csv file
-<div>
- <pre><tt>NY,New York,8143197 CA,Los Angeles,3844829 IL,Chicago,2842518 TX,Houston,2016582 PA,Philadelphia,1463281 AZ,Phoenix,1461575 TX,San Antonio,1256509 CA,San Diego,1255540 TX,Dallas,1213825 CA,San Jose,912332 </tt></pre>
-</div> * Create us_population_queries.sql file
-<div>
- <pre><tt>SELECT state as âStateâ,count(city) as âCity Countâ,sum(population) as âPopulation Sumâ FROM us_population GROUP BY state ORDER BY sum(population) DESC; </tt></pre>
-</div> * Execute the following command from a command terminal
-<div>
- <pre><tt>./psql.sh <your_zookeeper_quorum> us_population.sql us_population.csv us_population_queries.sql </tt></pre>
-</div>
+<p><i><b>I donât want to download and setup anything else!</b></i><br /> Ok, fair enough - you can create your own SQL scripts and execute them using our command line tool instead. Letâs walk through an example now. In the bin directory of your install location:</p>
+<ul>
+ <li>Create us_population.sql file
+ <div>
+ <pre><tt>CREATE TABLE IF NOT EXISTS us_population ( state CHAR(2) NOT NULL, city VARCHAR NOT NULL, population BIGINT CONSTRAINT my_pk PRIMARY KEY (state, city));</tt></pre>
+ </div></li>
+ <li>Create us_population.csv file
+ <div>
+ <pre><tt>NY,New York,8143197 CA,Los Angeles,3844829 IL,Chicago,2842518 TX,Houston,2016582 PA,Philadelphia,1463281 AZ,Phoenix,1461575 TX,San Antonio,1256509 CA,San Diego,1255540 TX,Dallas,1213825 CA,San Jose,912332 </tt></pre>
+ </div></li>
+ <li> <p>Create us_population_queries.sql file </p>
+ <div>
+ <pre><tt>SELECT state as âStateâ,count(city) as âCity Countâ,sum(population) as âPopulation Sumâ FROM us_population GROUP BY state ORDER BY sum(population) DESC; </tt></pre>
+ </div></li>
+ <li> <p>Execute the following command from a command terminal </p>
+ <div>
+ <pre><tt>./psql.sh <your_zookeeper_quorum> us_population.sql us_population.csv us_population_queries.sql </tt></pre>
+ </div></li>
+</ul>
<p>Congratulations! Youâve just created your first Phoenix table, inserted data into it, and executed an aggregate query with just a few lines of code in 15 minutes or less! </p>
<p><i><b>Big deal - 10 rows! What else you got?</b></i><br /> Ok, ok - tough crowd. Check out our <tt>bin/performance.sh</tt> script to create as many rows as you want, for any schema you come up with, and run timed queries against it.</p>
<p><i><b>Why is it called Phoenix anyway? Did some other project crash and burn and this is the next generation?</b></i><br /> Iâm sorry, but weâre out of time and space, so weâll have to answer that next time!</p>
Modified: incubator/phoenix/site/publish/building.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/building.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/building.html (original)
+++ incubator/phoenix/site/publish/building.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-11
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: incubator/phoenix/site/publish/download.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/download.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/download.html (original)
+++ incubator/phoenix/site/publish/download.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-14
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -153,7 +153,12 @@
<pre> $ psql.sh localhost ../examples/web_stat.sql ../examples/web_stat.csv ../examples/web_stat_queries.sql
</pre>
</div>
- <p>Other alternatives include: * Using our <a href="mr_dataload.html">map-reduce based CSV loader</a> for bigger data sets * <a href="index.html#Mapping-to-an-Existing-HBase-Table">Mapping an existing HBase table to a Phoenix table</a> and using the <a href="language/index.html#upsert_select">UPSERT SELECT</a> command to populate a new table. * Populating the table through our <a href="language/index.html#upsert_values">UPSERT VALUES</a> command.</p>
+ <p>Other alternatives include:</p>
+ <ul>
+ <li>Using our <a href="mr_dataload.html">map-reduce based CSV loader</a> for bigger data sets</li>
+ <li><a href="index.html#Mapping-to-an-Existing-HBase-Table">Mapping an existing HBase table to a Phoenix table</a> and using the <a href="language/index.html#upsert_select">UPSERT SELECT</a> command to populate a new table.</li>
+ <li>Populating the table through our <a href="language/index.html#upsert_values">UPSERT VALUES</a> command.</li>
+ </ul>
</div>
</div>
<div class="section">
Modified: incubator/phoenix/site/publish/language/datatypes.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/language/datatypes.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/language/datatypes.html (original)
+++ incubator/phoenix/site/publish/language/datatypes.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-11
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: incubator/phoenix/site/publish/language/functions.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/language/functions.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/language/functions.html (original)
+++ incubator/phoenix/site/publish/language/functions.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-11
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: incubator/phoenix/site/publish/language/index.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/language/index.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/language/index.html (original)
+++ incubator/phoenix/site/publish/language/index.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-11
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: incubator/phoenix/site/publish/mr_dataload.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/mr_dataload.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/mr_dataload.html (original)
+++ incubator/phoenix/site/publish/mr_dataload.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-14
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -164,9 +164,11 @@
</div>
<div class="section">
<h4 id="How_to_run">How to run?</h4>
- <p>1- Please make sure that Hadoop cluster is working correctly and you are able to run any job like <a class="externalLink" href="http://wiki.apache.org/hadoop/WordCount">this</a>. </p>
- <p>2- Copy latest phoenix-[version].jar to hadoop/lib folder on each node or add it to Hadoop classpath.</p>
- <p>3- Run the bulk loader job using the script /bin/csv-bulk-loader.sh as below:</p>
+ <ol style="list-style-type: decimal">
+ <li> <p>Please make sure that Hadoop cluster is working correctly and you are able to run any job like <a class="externalLink" href="http://wiki.apache.org/hadoop/WordCount">this</a>.</p></li>
+ <li> <p>Copy latest phoenix-[version].jar to hadoop/lib folder on each node or add it to Hadoop classpath.</p></li>
+ <li> <p>Run the bulk loader job using the script /bin/csv-bulk-loader.sh as below:</p></li>
+ </ol>
<div class="source">
<pre>./csv-bulk-loader.sh <option value>
Modified: incubator/phoenix/site/publish/performance.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/performance.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/performance.html (original)
+++ incubator/phoenix/site/publish/performance.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-14
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -114,7 +114,11 @@
<div class="page-header">
<h1>Performance</h1>
</div>
-<p>Phoenix follows the philosophy of <b>bringing the computation to the data</b> by using: * <b>coprocessors</b> to perform operations on the server-side thus minimizing client/server data transfer * <b>custom filters</b> to prune data as close to the source as possible In addition, to minimize any startup costs, Phoenix uses native HBase APIs rather than going through the map/reduce framework.</p>
+<p>Phoenix follows the philosophy of <b>bringing the computation to the data</b> by using:</p>
+<ul>
+ <li><b>coprocessors</b> to perform operations on the server-side thus minimizing client/server data transfer</li>
+ <li><b>custom filters</b> to prune data as close to the source as possible In addition, to minimize any startup costs, Phoenix uses native HBase APIs rather than going through the map/reduce framework.</li>
+</ul>
<div class="section">
<h2 id="Phoenix_vs_related_products">Phoenix vs related products</h2>
<p>Below are charts showing relative performance between Phoenix and some other related products.</p>
@@ -127,7 +131,7 @@
<h3 id="Phoenix_vs_Impala_running_over_HBase">Phoenix vs Impala (running over HBase)</h3>
<p><img src="images/PhoenixVsImpala.png" alt="Phoenix vs Impala" /></p>
<p>Query: select count(1) from table over 1M and 5M rows. Data is 3 narrow columns. Number of Region Server: 1 (Virtual Machine, HBase heap: 2GB, Processor: 2 cores @ 3.3GHz Xeon)</p>
- <p>***</p>
+ <hr />
</div>
</div>
<div class="section">
Modified: incubator/phoenix/site/publish/pig_integration.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/pig_integration.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/pig_integration.html (original)
+++ incubator/phoenix/site/publish/pig_integration.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-14
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -130,7 +130,7 @@ STORE A into 'hbase://CORE.ENTITY_HISTOR
<div class="section">
<h3 id="Gotchas">Gotchas</h3>
<p>It is advised that the upsert operation be idempotent. That is, trying to re-upsert data should not cause any inconsistencies. This is important in the case when a Pig job fails in process of writing to a Phoenix table. There is no notion of rollback (due to lack of transactions in HBase), and re-trying the upsert with PhoenixHBaseStorage must result in the same data in HBase table.</p>
- <p>For eg, letâs assume we are writing records n1â¦.n10 to HBase. If the job fails in the middle of this process, we are left in an inconsistent state where n1â¦.n7 made it to the phoenix tables but n8â¦.n10 were missed. If we retry the same operation, n1â¦.n7 would be re-upserted and n8â¦.n10 would be upserted this time.</p>
+ <p>For example, letâs assume we are writing records n1â¦n10 to HBase. If the job fails in the middle of this process, we are left in an inconsistent state where n1â¦n7 made it to the phoenix tables but n8â¦n10 were missed. If we retry the same operation, n1â¦n7 would be re-upserted and n8â¦n10 would be upserted this time.</p>
</div>
</div>
<div class="section">
Modified: incubator/phoenix/site/publish/secondary_indexing.html
URL: http://svn.apache.org/viewvc/incubator/phoenix/site/publish/secondary_indexing.html?rev=1568572&r1=1568571&r2=1568572&view=diff
==============================================================================
--- incubator/phoenix/site/publish/secondary_indexing.html (original)
+++ incubator/phoenix/site/publish/secondary_indexing.html Sat Feb 15 01:00:06 2014
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2014-02-03
+ Generated by Apache Maven Doxia at 2014-02-14
Rendered using Maven Reflow Skin 1.0.0 (http://andriusvelykis.github.com/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -171,7 +171,13 @@
<h2 id="Data_Guarantees_and_Failure_Management">Data Guarantees and Failure Management</h2>
<p>On successful return to the client, all data is guaranteed to be written to all interested indexes and the primary table. For each individual data row, updates are an all-or-nothing, with a small gap of being behind. From the perspective of a single client, it either thinks all-or-none of the update worked.</p>
<p>We maintain index update durability by adding the index updates to the Write-Ahead-Log (WAL) entry of the primary table row. Only after the WAL entry is successfully synced to disk do we attempt to make the index/primary table updates. We write the index updates in parallel by default, leading to very high throughput. If the server crashes while we are writing the index updates, we replay the all the index updates to the index tables in the WAL recovery process and rely on the idempotence of the updates to ensure correctness. Therefore, index tables are only every a single edit ahead of the primary table.</p>
- <p>Its important to note several points: * We <i>do not provide full transactions</i> so you could see the index table out of sync with the primary table. * As noted above, this is ok as we are only a very small bit ahead and out of sync for very short periods * Each data row and its index row(s) are guaranteed to to be written or lost - we never see partial updates * All data is first written to index tables before the primary table</p>
+ <p>Itâs important to note several points:</p>
+ <ul>
+ <li>We <i>do not provide full transactions</i> so you could see the index table out of sync with the primary table.</li>
+ <li>As noted above, this is ok as we are only a very small bit ahead and out of sync for very short periods</li>
+ <li>Each data row and its index row(s) are guaranteed to to be written or lost - we never see partial updates</li>
+ <li>All data is first written to index tables before the primary table</li>
+ </ul>
<div class="section">
<h3 id="Singular_Write_Path">Singular Write Path</h3>
<p>There is a single write path that guarantees the failure properties. All writes to the HRegion get intercepted by our coprocessor. We then build the index updates based on the pending update (or updates, in the case of the batch). These update are then appended to the WAL entry for the original update.</p>
@@ -201,66 +207,58 @@
<p>Out the box, indexing is pretty fast. However, to optimize for your particular environment and workload, there are several properties you can tune.</p>
<p>All the following parameters must be set in <tt>hbase-site.xml</tt> - they are true for the entire cluster and all index tables, as well as across all regions on the same server (so, for instance, a single server would not write to too many different index tables at once).</p>
<ol style="list-style-type: decimal">
- <li>index.builder.threads.max</li>
- </ol>
- <ul>
- <li>Number of threads to used to build the index update from the primary table update</li>
- <li>Increasing this value overcomes the bottleneck of reading the current row state from the underlying HRegion. Tuning this value too high will just bottleneck at the HRegion as it will not be able to handle too many concurrent scan requests as well as general thread-swapping concerns.</li>
- <li><b>Default: 10</b></li>
- </ul>
- <ol style="list-style-type: decimal">
- <li>index.builder.threads.keepalivetime</li>
- </ol>
- <ul>
- <li>Amount of time in seconds after we expire threads in the builder thread pool.</li>
- <li>Unused threads are immediately released after this amount of time and not core threads are retained (though this last is a small concern as tables are expected to sustain a fairly constant write load), but simultaneously allows us to drop threads if we are not seeing the expected load.</li>
- <li><b>Default: 60</b></li>
- </ul>
- <ol style="list-style-type: decimal">
- <li>index.writer.threads.max</li>
- </ol>
- <ul>
- <li>Number of threads to use when writing to the target index tables.</li>
- <li>The first level of parallelization, on a per-table basis - it should roughly correspond to the number of index tables</li>
- <li><b>Default: 10</b></li>
- </ul>
- <ol style="list-style-type: decimal">
- <li>index.writer.threads.keepalivetime</li>
- </ol>
- <ul>
- <li>Amount of time in seconds after we expire threads in the writer thread pool.</li>
- <li>Unused threads are immediately released after this amount of time and not core threads are retained (though this last is a small concern as tables are expected to sustain a fairly constant write load), but simultaneously allows us to drop threads if we are not seeing the expected load.</li>
- <li><b>Default: 60</b></li>
- </ul>
- <ol style="list-style-type: decimal">
- <li>hbase.htable.threads.max</li>
- </ol>
- <ul>
- <li>Number of threads each index HTable can use for writes.</li>
- <li>Increasing this allows more concurrent index updates (for instance across batches), leading to high overall throughput.</li>
- <li><b>Default: 2,147,483,647</b></li>
- </ul>
- <ol style="list-style-type: decimal">
- <li>hbase.htable.threads.keepalivetime</li>
+ <li> <p>index.builder.threads.max</p>
+ <ul>
+ <li>Number of threads to used to build the index update from the primary table update</li>
+ <li>Increasing this value overcomes the bottleneck of reading the current row state from the underlying HRegion. Tuning this value too high will just bottleneck at the HRegion as it will not be able to handle too many concurrent scan requests as well as general thread-swapping concerns.</li>
+ <li><b>Default: 10</b></li>
+ </ul></li>
+ <li> <p>index.builder.threads.keepalivetime</p>
+ <ul>
+ <li>Amount of time in seconds after we expire threads in the builder thread pool.</li>
+ <li>Unused threads are immediately released after this amount of time and not core threads are retained (though this last is a small concern as tables are expected to sustain a fairly constant write load), but simultaneously allows us to drop threads if we are not seeing the expected load.</li>
+ <li><b>Default: 60</b></li>
+ </ul></li>
+ <li> <p>index.writer.threads.max</p>
+ <ul>
+ <li>Number of threads to use when writing to the target index tables.</li>
+ <li>The first level of parallelization, on a per-table basis - it should roughly correspond to the number of index tables</li>
+ <li><b>Default: 10</b></li>
+ </ul></li>
+ <li> <p>index.writer.threads.keepalivetime</p>
+ <ul>
+ <li>Amount of time in seconds after we expire threads in the writer thread pool.</li>
+ <li>Unused threads are immediately released after this amount of time and not core threads are retained (though this last is a small concern as tables are expected to sustain a fairly constant write load), but simultaneously allows us to drop threads if we are not seeing the expected load.</li>
+ <li><b>Default: 60</b></li>
+ </ul></li>
+ <li> <p>hbase.htable.threads.max</p>
+ <ul>
+ <li>Number of threads each index HTable can use for writes.</li>
+ <li>Increasing this allows more concurrent index updates (for instance across batches), leading to high overall throughput.</li>
+ <li><b>Default: 2,147,483,647</b></li>
+ </ul></li>
+ <li> <p>hbase.htable.threads.keepalivetime</p>
+ <ul>
+ <li>Amount of time in seconds after we expire threads in the HTableâs thread pool.</li>
+ <li>Using the âdirect handoffâ approach, new threads will only be created if it is necessary and will grow unbounded. This could be bad but HTables only create as many Runnables as there are region servers; therefore, it also scales when new region servers are added.</li>
+ <li><b>Default: 60</b></li>
+ </ul></li>
+ <li> <p>index.tablefactory.cache.size</p>
+ <ul>
+ <li>Number of index HTables we should keep in cache.</li>
+ <li>Increasing this number ensures that we do not need to recreate an HTable for each attempt to write to an index table. Conversely, you could see memory pressure if this value is set too high.</li>
+ <li><b>Default: 10</b></li>
+ </ul></li>
</ol>
- <ul>
- <li>Amount of time in seconds after we expire threads in the HTableâs thread pool.</li>
- <li>Using the âdirect handoffâ approach, new threads will only be created if it is necessary and will grow unbounded. This could be bad but HTables only create as many Runnables as there are region servers; therefore, it also scales when new region servers are added.</li>
- <li><b>Default: 60</b></li>
- </ul>
- <ol style="list-style-type: decimal">
- <li>index.tablefactory.cache.size</li>
- </ol>
- <ul>
- <li>Number of index HTables we should keep in cache.</li>
- <li>Increasing this number ensures that we do not need to recreate an HTable for each attempt to write to an index table. Conversely, you could see memory pressure if this value is set too high.</li>
- <li><b>Default: 10</b></li>
- </ul>
<h1>Performance</h1>
<p>We track secondary index performance via our <a class="externalLink" href="http://phoenix-bin.github.io/client/performance/latest.htm">performance framework</a>. This is a generic test of performance based on defaults - your results will vary based on hardware specs as well as you individual configuration.</p>
<p>That said, we have seen secondary indexing (both immutable and mutable) go as quickly as < 2x the regular write path on a small, (3 node) desktop-based cluster. This is actually a phenomenal as we have to write to multiple tables as well as build the index update.</p>
<h1>Presentations</h1>
- <p>There have been several presentations given on how secondary indexing works in Phoenix that have a more indepth look at how indexing works (with pretty pictures!): * <a class="externalLink" href="http://files.meetup.com/1350427/PhoenixIndexing-SF-HUG_09-26-13.pptx">San Francisco HBase Meetup</a> - Sept. 26, 2013 * <a class="externalLink" href="http://www.slideshare.net/jesse_yates/phoenix-secondary-indexing-la-hug-sept-9th-2013">Los Anglees HBase Meetup</a> - Sept, 4th, 2013</p>
+ <p>There have been several presentations given on how secondary indexing works in Phoenix that have a more indepth look at how indexing works (with pretty pictures!):</p>
+ <ul>
+ <li><a class="externalLink" href="http://files.meetup.com/1350427/PhoenixIndexing-SF-HUG_09-26-13.pptx">San Francisco HBase Meetup</a> - Sept. 26, 2013</li>
+ <li><a class="externalLink" href="http://www.slideshare.net/jesse_yates/phoenix-secondary-indexing-la-hug-sept-9th-2013">Los Anglees HBase Meetup</a> - Sept, 4th, 2013</li>
+ </ul>
</div>
</div>
</div>