You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@phoenix.apache.org by ra...@apache.org on 2016/08/12 13:12:52 UTC
svn commit: r1756181 - in /phoenix/site: publish/release_notes.html
publish/secondary_indexing.html source/src/site/markdown/release_notes.md
source/src/site/markdown/secondary_indexing.md
Author: rajeshbabu
Date: Fri Aug 12 13:12:51 2016
New Revision: 1756181
URL: http://svn.apache.org/viewvc?rev=1756181&view=rev
Log:
Updating docs and release notes for new local index implementation(Rajeshbabu)
Modified:
phoenix/site/publish/release_notes.html
phoenix/site/publish/secondary_indexing.html
phoenix/site/source/src/site/markdown/release_notes.md
phoenix/site/source/src/site/markdown/secondary_indexing.md
Modified: phoenix/site/publish/release_notes.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/release_notes.html?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/publish/release_notes.html (original)
+++ phoenix/site/publish/release_notes.html Fri Aug 12 13:12:51 2016
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2016-08-10
+ Generated by Apache Maven Doxia at 2016-08-11
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -155,6 +155,22 @@
<h3 id="Phoenix-4.8.0_Release_Notes"><u>Phoenix-4.8.0 Release Notes</u></h3>
<p><a class="externalLink" href="https://issues.apache.org/jira/browse/PHOENIX-3164">PHOENIX-3164</a> is a relatively serious bug that affects the <a class="externalLink" href="http://phoenix.apache.org/server.html">Phoenix Query Server</a> deployed with “security enabled” (Kerberos or Active Directory). Due to another late-game change in the 4.8.0 release as well as an issue with the use of Hadoop’s UserGroupInformation class, every “client session” to the Phoenix Query Server with security enabled will result in a new instance of the Phoenix JDBC driver <tt>PhoenixConnection</tt> (and other related classes). This ultimately results in a new connection to ZooKeeper for each “client session”.</p>
<p>Within a short amount of time of active use with the Phoenix Query Server creating a new ZooKeeper connection for each “client session”, the number of ZooKeeper connections will have grown rapidly likely triggering ZooKeeper’s built-in denial of service protection (<a class="externalLink" href="https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html">maxClientCnxns</a>). This will cause all future connections to ZooKeeper by the host running the Phoenix Query Server to be dropped. This would prevent all HBase client API calls which need to access ZooKeeper from completing.</p>
+ <p>As part of <a class="externalLink" href="https://issues.apache.org/jira/browse/PHOENIX-1734">PHOENIX-1734</a> we have changed the local index implementation to store index data in the separate column families in the same data table. So while upgrading the phoenix at server we need to remove below local index related configurations from <tt>hbase-site.xml</tt> and run upgrade steps mentioned <a href="secondary_indexing.html#Upgrading_Local_Indexes_created_before_4.8.0">here</a></p>
+ <div class="source">
+ <pre><property>
+ <name>hbase.master.loadbalancer.class</name>
+ <value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
+</property>
+<property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
+</property>
+<property>
+ <name>hbase.coprocessor.regionserver.classes</name>
+ <value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
+</property>
+</pre>
+ </div>
</div>
<div class="section">
<h3 id="Phoenix-4.5.0_Release_Notes"><u>Phoenix-4.5.0 Release Notes</u></h3>
Modified: phoenix/site/publish/secondary_indexing.html
URL: http://svn.apache.org/viewvc/phoenix/site/publish/secondary_indexing.html?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/publish/secondary_indexing.html (original)
+++ phoenix/site/publish/secondary_indexing.html Fri Aug 12 13:12:51 2016
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2016-08-11
+ Generated by Apache Maven Doxia at 2016-08-12
Rendered using Reflow Maven Skin 1.1.0 (http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -180,7 +180,7 @@
</div>
<div class="section">
<h2 id="Local_Indexes">Local Indexes</h2>
- <p>Local indexing targets <i>write heavy</i>, <i>space constrained</i> use cases. Just like with global indexes, Phoenix will automatically select whether or not to use a local index at query-time. With local indexes, index data and table data co-reside on same server preventing any network overhead during writes. Local indexes can be used even when the query isn’t fully covered (i.e. Phoenix automatically retrieve the columns not in the index through point gets against the data table). Unlike global indexes, all local indexes of a table are stored in a single, separate shared table. At read time when the local index is used, every region must be examined for the data as the exact region location of index data cannot be predetermined. Thus some overhead occurs at read-time.</p>
+ <p>Local indexing targets <i>write heavy</i>, <i>space constrained</i> use cases. Just like with global indexes, Phoenix will automatically select whether or not to use a local index at query-time. With local indexes, index data and table data co-reside on same server preventing any network overhead during writes. Local indexes can be used even when the query isn’t fully covered (i.e. Phoenix automatically retrieve the columns not in the index through point gets against the data table). Unlike global indexes, all local indexes of a table are stored in a single, separate shared table prior to 4.8.0 version. From 4.8.0 onwards we are storing all local index data in the separate shadow column families in the same data table. At read time when the local index is used, every region must be examined for the data as the exact region location of index data cannot be predetermined. Thus some overhead occurs at read-time.</p>
</div>
<div class="section">
<h2 id="Index_Population">Index Population</h2>
@@ -350,8 +350,7 @@ CREATE LOCAL INDEX my_index ON my_table
</pre>
</div>
<p>The above properties prevent deadlocks from occurring during index maintenance for global indexes (HBase 0.98.4+ and Phoenix 4.3.1+ only) by ensuring index updates are processed with a higher priority than data updates. It also prevents deadlocks by ensuring metadata rpc calls are processed with a higher priority than data rpc calls.</p>
- <p>Local indexing also requires special configurations in the master to ensure data table and local index regions co-location.</p>
- <p>You will need to add the following parameters to <tt>hbase-site.xml</tt> on the master:</p>
+ <p>From Phoenix 4.8.0 onward, no configuration changes are required to use local indexing. In Phoenix 4.7 and below, the following configuration changes are required to the server-side hbase-site.xml on the master and regions server nodes:</p>
<div class="source">
<pre><property>
<name>hbase.master.loadbalancer.class</name>
@@ -361,16 +360,25 @@ CREATE LOCAL INDEX my_index ON my_table
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
-</pre>
- </div>
- <p>To support local index regions merge on data regions merge you will need to add the following parameter to <tt>hbase-site.xml</tt> in all the region servers and restart. (It’s applicable for Phoenix 4.3+ versions)</p>
- <div class="source">
- <pre><property>
+<property>
<name>hbase.coprocessor.regionserver.classes</name>
<value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
</property>
</pre>
</div>
+ <div class="section">
+ <h3 id="Upgrading_Local_Indexes_created_before_4.8.0">Upgrading Local Indexes created before 4.8.0</h3>
+ <p>While upgrading the Phoenix to 4.8.0+ version at server remove above three local indexing related configurations from <tt>hbase-site.xml</tt> if present. From client we are supporting both online(while initializing the connection from phoenix client of 4.8.0+ versions) and offline(using psql tool) upgrade of local indexes created before 4.8.0. As part of upgrade we recreate the local indexes in ASYNC mode. After upgrade user need to build the indexes using <a class="externalLink" href="http://phoenix.apache.org/secondary_indexing.html#Index_Population">IndexTool</a></p>
+ <p>Following client side configuration used in the upgrade. </p>
+ <ol style="list-style-type: decimal">
+ <li><tt>phoenix.client.localIndexUpgrade</tt>
+ <ul>
+ <li>The value of it is true means online upgrade and false means offline upgrade.</li>
+ <li><b>Default: true</b></li>
+ </ul></li>
+ </ol>
+ <p>Command to run offline upgrade using psql tool <tt>$ psql [zookeeper] -l</tt></p>
+ </div>
</div>
<div class="section">
<h2 id="Tuning">Tuning</h2>
Modified: phoenix/site/source/src/site/markdown/release_notes.md
URL: http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/release_notes.md?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/release_notes.md (original)
+++ phoenix/site/source/src/site/markdown/release_notes.md Fri Aug 12 13:12:51 2016
@@ -22,6 +22,27 @@ will cause all future connections to Zoo
be dropped. This would prevent all HBase client API calls which need to access ZooKeeper
from completing.
+As part of [PHOENIX-1734](https://issues.apache.org/jira/browse/PHOENIX-1734) we have changed
+the local index implementation to store index data in the separate column families in the same
+ data table. So while upgrading the phoenix at server we need to remove below local index
+related configurations from `hbase-site.xml` and run upgrade steps mentioned
+[here](secondary_indexing.html#Upgrading_Local_Indexes_created_before_4.8.0)
+
+```
+<property>
+ <name>hbase.master.loadbalancer.class</name>
+ <value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
+</property>
+<property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
+</property>
+<property>
+ <name>hbase.coprocessor.regionserver.classes</name>
+ <value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
+</property>
+```
+
###<u>Phoenix-4.5.0 Release Notes</u>
Both [PHOENIX-2067](https://issues.apache.org/jira/browse/PHOENIX-2067) and
[PHOENIX-2120](https://issues.apache.org/jira/browse/PHOENIX-2120) cause rows to not be ordered
Modified: phoenix/site/source/src/site/markdown/secondary_indexing.md
URL: http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/secondary_indexing.md?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/secondary_indexing.md (original)
+++ phoenix/site/source/src/site/markdown/secondary_indexing.md Fri Aug 12 13:12:51 2016
@@ -37,7 +37,7 @@ Each are useful in different scenarios a
Global indexing targets _read heavy_ uses cases. With global indexes, all the performance penalties for indexes occur at write time. We intercept the data table updates on write ([DELETE](language/index.html#delete), [UPSERT VALUES](language/index.html#upsert_values) and [UPSERT SELECT](language/index.html#upsert_select)), build the index update and then sent any necessary updates to all interested index tables. At read time, Phoenix will select the index table to use that will produce the fastest query time and directly scan it just like any other HBase table. By default, unless hinted, an index will not be used for a query that references a column that isn't part of the index.
## Local Indexes
-Local indexing targets _write heavy_, _space constrained_ use cases. Just like with global indexes, Phoenix will automatically select whether or not to use a local index at query-time. With local indexes, index data and table data co-reside on same server preventing any network overhead during writes. Local indexes can be used even when the query isn't fully covered (i.e. Phoenix automatically retrieve the columns not in the index through point gets against the data table). Unlike global indexes, all local indexes of a table are stored in a single, separate shared table. At read time when the local index is used, every region must be examined for the data as the exact region location of index data cannot be predetermined. Thus some overhead occurs at read-time.
+Local indexing targets _write heavy_, _space constrained_ use cases. Just like with global indexes, Phoenix will automatically select whether or not to use a local index at query-time. With local indexes, index data and table data co-reside on same server preventing any network overhead during writes. Local indexes can be used even when the query isn't fully covered (i.e. Phoenix automatically retrieve the columns not in the index through point gets against the data table). Unlike global indexes, all local indexes of a table are stored in a single, separate shared table prior to 4.8.0 version. From 4.8.0 onwards we are storing all local index data in the separate shadow column families in the same data table. At read time when the local index is used, every region must be examined for the data as the exact region location of index data cannot be predetermined. Thus some overhead occurs at read-time.
## Index Population
By default, when an index is created, it is populated synchronously during the CREATE INDEX call. This may not be feasible depending on the current size of the data table. As of 4.5, initially population of an index may be done asynchronously by including the ASYNC keyword in the index creation DDL statement:
@@ -232,9 +232,7 @@ The above property enables custom WAL ed
The above properties prevent deadlocks from occurring during index maintenance for global indexes (HBase 0.98.4+ and Phoenix 4.3.1+ only) by ensuring index updates are processed with a higher priority than data updates. It also prevents deadlocks by ensuring metadata rpc calls are processed with a higher priority than data rpc calls.
-Local indexing also requires special configurations in the master to ensure data table and local index regions co-location.
-
-You will need to add the following parameters to `hbase-site.xml` on the master:
+From Phoenix 4.8.0 onward, no configuration changes are required to use local indexing. In Phoenix 4.7 and below, the following configuration changes are required to the server-side hbase-site.xml on the master and regions server nodes:
```
<property>
@@ -245,16 +243,22 @@ You will need to add the following param
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
-```
-
-To support local index regions merge on data regions merge you will need to add the following parameter to `hbase-site.xml` in all the region servers and restart. (It’s applicable for Phoenix 4.3+ versions)
-
-```
<property>
<name>hbase.coprocessor.regionserver.classes</name>
<value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
</property>
```
+### Upgrading Local Indexes created before 4.8.0
+While upgrading the Phoenix to 4.8.0+ version at server remove above three local indexing related configurations from `hbase-site.xml` if present. From client we are supporting both online(while initializing the connection from phoenix client of 4.8.0+ versions) and offline(using psql tool) upgrade of local indexes created before 4.8.0. As part of upgrade we recreate the local indexes in ASYNC mode. After upgrade user need to build the indexes using [IndexTool](http://phoenix.apache.org/secondary_indexing.html#Index_Population)
+
+Following client side configuration used in the upgrade.
+
+1. <code>phoenix.client.localIndexUpgrade</code>
+ * The value of it is true means online upgrade and false means offline upgrade.
+ * **Default: true**
+
+Command to run offline upgrade using psql tool
+`$ psql [zookeeper] -l`
## Tuning
Out the box, indexing is pretty fast. However, to optimize for your particular environment and workload, there are several properties you can tune.