svn commit: r1181378 - /hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java

[ns...@apache.org]

Author: nspiegelberg
Date: Tue Oct 11 02:03:37 2011
New Revision: 1181378

URL: http://svn.apache.org/viewvc?rev=1181378&view=rev
Log:
HBASE-3006: Fix for severe read penalty on compressed column families

Summary:
Details up in:
https://issues.apache.org/jira/browse/HBASE-3006

Summary: Cuts down RPC calls to DFS by 5-10x when a HBase block cache miss
happens, and it needs to go to DFS.

Test Plan:
Ran Pranav's schema search test, and for loading full index for a user:

Averages dropped from 1-2 seconds to 100ms for my test runs. No outliers > 10
seconds. Previously we had many, and some which took 40-50 seconds.

For 1-letter prefix match: the averages dropped from 25-50 ms to under 5 ms.
[Caveat: In this run about 50% of the users were already warmed by the first
test.]

Running unit tests now.

DiffCamp Revision: 158894
Reviewed By: nspiegelberg
Commenters: pmalik
CC: nspiegelberg, pmalik, jfan, kannan, tnovak, ruifang, hbase@lists
Tasks:
#383346: Clicking other folder takes forever then displays no messages when I
have 100's of messages.

Revert Plan:
OK

Modified:
    hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java

Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java?rev=1181378&r1=1181377&r2=1181378&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java Tue Oct 11 02:03:37 2011
@@ -19,6 +19,7 @@
  */
 package org.apache.hadoop.hbase.io.hfile;
 
+import java.io.BufferedInputStream;
 import java.io.Closeable;
 import java.io.DataInputStream;
 import java.io.DataOutputStream;
@@ -1090,9 +1091,11 @@ public class HFile {
         // bunch of data w/o regard to whether decompressor is coming to end of a
         // decompression.
         InputStream is = this.compressAlgo.createDecompressionStream(
-          new BoundedRangeFileInputStream(this.istream, offset, compressedSize,
-            pread),
-          decompressor, 0);
+            new BufferedInputStream(
+                new BoundedRangeFileInputStream(this.istream, offset, compressedSize,
+                                                pread),
+                Math.min(65536, compressedSize)),
+            decompressor, 0);
         buf = ByteBuffer.allocate(decompressedSize);
         IOUtils.readFully(is, buf.array(), 0, buf.capacity());
         is.close();