You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by en...@apache.org on 2013/03/30 01:19:57 UTC
svn commit: r1462679 [14/14] - in /hbase/hbase.apache.org/trunk: ./ book/
case_studies/ community/ configuration/ developer/ getting_started/
ops_mgt/ performance/ rpc/
Added: hbase/hbase.apache.org/trunk/rpc/apas03.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/rpc/apas03.html?rev=1462679&view=auto
==============================================================================
--- hbase/hbase.apache.org/trunk/rpc/apas03.html (added)
+++ hbase/hbase.apache.org/trunk/rpc/apas03.html Sat Mar 30 00:19:55 2013
@@ -0,0 +1,50 @@
+<html><head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ <title>A.3. RPC</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="rpc.html" title="Appendix A. 0.95 RPC Specification"><link rel="up" href="rpc.html" title="Appendix A. 0.95 RPC Specification"><link rel="prev" href="apas02.html" title="A.2. TODO"><link rel="next" href="apas04.html" title="A.4. Notes"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">A.3. RPC</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apas02.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="apas04.html">Next</a></td></tr></table><hr></div><div class="section" title="A.3. RPC"><div class="titlepage"><div><
div><h2 class="title" style="clear: both"><a name="d551e51"></a>A.3. RPC</h2></div></div></div><p>The client will send setup information on connection establish.
+ Thereafter, the client invokes methods against the remote server sending a protobuf Message and receiving a protobuf Message in response.
+ Communication is synchronous. All back and forth is preceded by an int that has the total length of the request/response.
+ Optionally, Cells(KeyValues) can be passed outside of protobufs in follow-behind Cell blocks (because
+ <a class="link" href="https://docs.google.com/document/d/1WEtrq-JTIUhlnlnvA0oYRLp0F8MKpEBeBSCFcQiacdw/edit#" target="_top">we can’t protobuf megabytes of KeyValues</a> or Cells).
+ These CellBlocks are encoded and optionally compressed.</p><p></p><p>For more detail on the protobufs involved, see the
+ <a class="link" href="http://svn.apache.org/viewvc/hbase/trunk/hbase-protocol/src/main/protobuf/RPC.proto?view=markup" target="_top">RPC.proto</a> file in trunk.</p><div class="section" title="A.3.1. Connection Setup"><div class="titlepage"><div><div><h3 class="title"><a name="d551e65"></a>A.3.1. Connection Setup</h3></div></div></div><p>Client initiates connection.</p><div class="section" title="A.3.1.1. Client"><div class="titlepage"><div><div><h4 class="title"><a name="d551e70"></a>A.3.1.1. Client</h4></div></div></div><p>On connection setup, client sends a preamble followed by a connection header.
+ </p><div class="section" title="A.3.1.1.1. <preamble>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e75"></a>A.3.1.1.1. <preamble></h5></div></div></div><pre class="programlisting"><MAGIC 4 byte integer> <1 byte RPC Format Version> <1 byte auth type><sup>[<a name="d551e81" href="#ftn.d551e81" class="footnote">1</a>]</sup></pre><p>E.g.: HBas0x000x80 -- 4 bytes of MAGIC -- ‘HBas’ -- plus one-byte of version, 0 in this case, and one byte, 0x80 (SIMPLE). of an auth type.</p></div><div class="section" title="A.3.1.1.2. <Protobuf ConnectionHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e86"></a>A.3.1.1.2. <Protobuf ConnectionHeader Message></h5></div></div></div><p>Has user info, and “protocol”, as well as the encoders and compression the client will use sending CellBlocks.
+ CellBlock encoders and compressors are for the life of the connection.
+ CellBlock encoders implement org.apache.hadoop.hbase.codec.Codec.
+ CellBlocks may then also be compressed.
+ Compressors implement org.apache.hadoop.io.compress.CompressionCodec.
+ This protobuf is written using writeDelimited so is prefaced by a pb varint
+ with its serialized length</p></div></div><div class="section" title="A.3.1.2. Server"><div class="titlepage"><div><div><h4 class="title"><a name="d551e92"></a>A.3.1.2. Server</h4></div></div></div><p>After client sends preamble and connection header,
+ server does NOT respond if successful connection setup.
+ No response means server is READY to accept requests and to give out response.
+ If the version or authentication in the preamble is not agreeable or the server has trouble parsing the preamble,
+ it will throw a org.apache.hadoop.hbase.ipc.FatalConnectionException explaining the error and will then disconnect.
+ If the client in the connection header -- i.e. the protobuf’d Message that comes after the connection preamble -- asks for for a
+ Service the server does not support or a codec the server does not have, again we throw a FatalConnectionException with explanation.</p></div></div><div class="section" title="A.3.2. Request"><div class="titlepage"><div><div><h3 class="title"><a name="d551e97"></a>A.3.2. Request</h3></div></div></div><p>After a Connection has been set up, client makes requests. Server responds.</p><p>A request is made up of a protobuf RequestHeader followed by a protobuf Message parameter.
+ The header includes the method name and optionally, metadata on the optional CellBlock that may be following.
+ The parameter type suits the method being invoked: i.e. if we are doing a getRegionInfo request,
+ the protobuf Message param will be an instance of GetRegionInfoRequest.
+ The response will be a GetRegionInfoResponse.
+ The CellBlock is optionally used ferrying the bulk of the RPC data: i.e Cells/KeyValues.</p><p></p><div class="section" title="A.3.2.1. Request Parts"><div class="titlepage"><div><div><h4 class="title"><a name="d551e105"></a>A.3.2.1. Request Parts</h4></div></div></div><div class="section" title="A.3.2.1.1. <Total Length>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e108"></a>A.3.2.1.1. <Total Length></h5></div></div></div><p>The request is prefaced by an int that holds the total length of what follows.</p></div><div class="section" title="A.3.2.1.2. <Protobuf RequestHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e113"></a>A.3.2.1.2. <Protobuf RequestHeader Message></h5></div></div></div><p>Will have call.id, trace.id, and method name, etc. including optional Metadata on the Cell block IFF one is following.
+ Data is protobuf’d inline in this pb Message or optionally comes in the following CellBlock</p></div><div class="section" title="A.3.2.1.3. <Protobuf Param Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e118"></a>A.3.2.1.3. <Protobuf Param Message></h5></div></div></div><p>If the method being invoked is getRegionInfo, if you study the Service descriptor for the client to regionserver protocol,
+ you will find that the request sends a GetRegionInfoRequest protobuf Message param in this position.</p></div><div class="section" title="A.3.2.1.4. <CellBlock>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e123"></a>A.3.2.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section" title="A.3.3. Response"><div class="titlepage"><div><div><h3 class="title"><a name="d551e130"></a>A.3.3. Response</h3></div></div></div><p>Same as Request, it is a protobuf ResponseHeader followed by a protobuf Message response where the Message response type suits the method invoked.
+ Bulk of the data may come in a following CellBlock.</p><div class="section" title="A.3.3.1. Response Parts"><div class="titlepage"><div><div><h4 class="title"><a name="d551e135"></a>A.3.3.1. Response Parts</h4></div></div></div><div class="section" title="A.3.3.1.1. <Total Length>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e138"></a>A.3.3.1.1. <Total Length></h5></div></div></div><p>The response is prefaced by an int that holds the total length of what follows.</p></div><div class="section" title="A.3.3.1.2. <Protobuf ResponseHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e143"></a>A.3.3.1.2. <Protobuf ResponseHeader Message></h5></div></div></div><p>Will have call.id, etc. Will include exception if failed processing. Optionally includes metadata on optional, IFF there is a CellBlock following.</p></div><div class="section" title="A.3.3.1.3. <Protobuf
Response Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e148"></a>A.3.3.1.3. <Protobuf Response Message></h5></div></div></div><p>Return or may be nothing if exception. If the method being invoked is getRegionInfo, if you study the Service descriptor for the client to regionserver protocol,
+ you will find that the response sends a GetRegionInfoResponse protobuf Message param in this position.</p></div><div class="section" title="A.3.3.1.4. <CellBlock>"><div class="titlepage"><div><div><h5 class="title"><a name="d551e153"></a>A.3.3.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section" title="A.3.4. Exceptions"><div class="titlepage"><div><div><h3 class="title"><a name="d551e160"></a>A.3.4. Exceptions</h3></div></div></div><p>There are two distinct types.
+ There is the request failed which is encapsulated inside the response header for the response.
+ The connection stays open to receive new requests.
+ The second type, the FatalConnectionException, kills the connection.</p><p>Exceptions can carry extra information.
+ See the ExceptionResponse protobuf type.
+ It has a flag to indicate do-no-retry as well as other miscellaneous payload to help improve client responsiveness.</p></div><div class="section" title="A.3.5. CellBlocks"><div class="titlepage"><div><div><h3 class="title"><a name="d551e167"></a>A.3.5. CellBlocks</h3></div></div></div><p>These are not versioned.
+ Server can do the codec or it cannot.
+ If new version of a codec with say, tighter encoding, then give it a new class name.
+ Codecs will live on the server for all time so old clients can connect.</p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d551e81" href="#d551e81" class="para">1</a>] </sup> We need the auth method spec. here so the connection header is encoded if auth enabled.</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
+ var disqus_url = 'http://hbase.apache.org/book';
+ var disqus_identifier = '';
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="apas02.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> <a accesskey="n" href="apas04.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">A.2. TODO </td><td width="20%" align="center"><a accesskey="h" href="rpc.html">Home</a></td><td width="40%" align="right" valign="top"> A.4. Notes</td></tr></table></div></body></html>
\ No newline at end of file
Added: hbase/hbase.apache.org/trunk/rpc/apas04.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/rpc/apas04.html?rev=1462679&view=auto
==============================================================================
--- hbase/hbase.apache.org/trunk/rpc/apas04.html (added)
+++ hbase/hbase.apache.org/trunk/rpc/apas04.html Sat Mar 30 00:19:55 2013
@@ -0,0 +1,17 @@
+<html><head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ <title>A.4. Notes</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="rpc.html" title="Appendix A. 0.95 RPC Specification"><link rel="up" href="rpc.html" title="Appendix A. 0.95 RPC Specification"><link rel="prev" href="apas03.html" title="A.3. RPC"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">A.4. Notes</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apas03.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> </td></tr></table><hr></div><div class="section" title="A.4. Notes"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d551e172"></a>A.4. Notes</h2></div></div></
div><div class="section" title="A.4.1. Constraints"><div class="titlepage"><div><div><h3 class="title"><a name="d551e175"></a>A.4.1. Constraints</h3></div></div></div><p>In some part, current wire-format -- i.e. all requests and responses preceeded by a length -- has been dictated by current server non-async architecture.</p></div><div class="section" title="A.4.2. One fat pb request or header+param"><div class="titlepage"><div><div><h3 class="title"><a name="d551e180"></a>A.4.2. One fat pb request or header+param</h3></div></div></div><p>We went with pb header followed by pb param making a request and a pb header followed by pb response for now.
+ Doing header+param rather than a single protobuf Message with both header and param content:</p><p>
+ </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Is closer to what we currently have</p></li><li class="listitem"><p>Having a single fat pb requires extra copying putting the already pb’d param into the body of the fat request pb (and same making result)</p></li><li class="listitem"><p>We can decide whether to accept the request or not before we read the param; for example, the request might be low priority. As is, we read header+param in one go as server is currently implemented so this is a TODO.</p></li></ol></div><p>
+ </p><p>The advantages are minor. If later, fat request has clear advantage, can roll out a v2 later.</p></div><div class="section" title="A.4.3. Compression"><div class="titlepage"><div><div><h3 class="title"><a name="d551e200"></a>A.4.3. Compression</h3></div></div></div><p>Uses hadoops compression codecs.</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
+ var disqus_url = 'http://hbase.apache.org/book';
+ var disqus_identifier = '';
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="apas03.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> </td></tr><tr><td width="40%" align="left" valign="top">A.3. RPC </td><td width="20%" align="center"><a accesskey="h" href="rpc.html">Home</a></td><td width="40%" align="right" valign="top"> </td></tr></table></div></body></html>
\ No newline at end of file
Modified: hbase/hbase.apache.org/trunk/source-repository.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/source-repository.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/source-repository.html (original)
+++ hbase/hbase.apache.org/trunk/source-repository.html Sat Mar 30 00:19:55 2013
@@ -1,6 +1,6 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia at Mar 21, 2013 -->
+<!-- Generated by Apache Maven Doxia at Mar 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -12,7 +12,7 @@
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<link rel="shortcut icon" href="/images/favicon.ico" />
- <meta name="Date-Revision-yyyymmdd" content="20130321" />
+ <meta name="Date-Revision-yyyymmdd" content="20130329" />
<meta http-equiv="Content-Language" content="en" />
<!--Google Analytics-->
<script type="text/javascript">
@@ -185,7 +185,7 @@ http-proxy-port = 3128
<div id="footer">
<div class="xright">
- <span id="publishDate">Last Published: 2013-03-21</span>
+ <span id="publishDate">Last Published: 2013-03-29</span>
| <span id="projectVersion">Version: 0.97-SNAPSHOT</span>
</div>
Modified: hbase/hbase.apache.org/trunk/sponsors.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/sponsors.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/sponsors.html (original)
+++ hbase/hbase.apache.org/trunk/sponsors.html Sat Mar 30 00:19:55 2013
@@ -1,6 +1,6 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia at Mar 21, 2013 -->
+<!-- Generated by Apache Maven Doxia at Mar 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -13,7 +13,7 @@
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<link rel="shortcut icon" href="/images/favicon.ico" />
- <meta name="Date-Revision-yyyymmdd" content="20130321" />
+ <meta name="Date-Revision-yyyymmdd" content="20130329" />
<meta http-equiv="Content-Language" content="en" />
<!--Google Analytics-->
<script type="text/javascript">
@@ -216,7 +216,7 @@
<div id="footer">
<div class="xright">
- <span id="publishDate">Last Published: 2013-03-21</span>
+ <span id="publishDate">Last Published: 2013-03-29</span>
| <span id="projectVersion">Version: 0.97-SNAPSHOT</span>
</div>
Modified: hbase/hbase.apache.org/trunk/team-list.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/team-list.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/team-list.html (original)
+++ hbase/hbase.apache.org/trunk/team-list.html Sat Mar 30 00:19:55 2013
@@ -1,6 +1,6 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia at Mar 21, 2013 -->
+<!-- Generated by Apache Maven Doxia at Mar 29, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -12,7 +12,7 @@
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<link rel="shortcut icon" href="/images/favicon.ico" />
- <meta name="Date-Revision-yyyymmdd" content="20130321" />
+ <meta name="Date-Revision-yyyymmdd" content="20130329" />
<meta http-equiv="Content-Language" content="en" />
<!--Google Analytics-->
<script type="text/javascript">
@@ -226,7 +226,7 @@ window.onLoad = init();
<div id="footer">
<div class="xright">
- <span id="publishDate">Last Published: 2013-03-21</span>
+ <span id="publishDate">Last Published: 2013-03-29</span>
| <span id="projectVersion">Version: 0.97-SNAPSHOT</span>
</div>