[jira] [Comment Edited] (HBASE-20218) Proposed Performance Enhancements For TableSnapshotInputFomat

["Saad Mufti (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404235#comment-16404235 ] 

Saad Mufti edited comment on HBASE-20218 at 3/18/18 11:14 PM:
--------------------------------------------------------------

{quote}You put the above in Release Note field.
{quote}
Removed.
{quote}
{code:java}
+  public static final String PREFETCH_BLOCKS_ON_OPEN_OVERRIDE_KEY =
+    "hbase.rs.prefetchblocksonopen.override";
{code}
There is config, "hbase.rs.prefetchblocksonopen" already. Have you tried overriding value for this config ?
{quote}
hbase.rs.prefetchblocksonopen if set to true, overrides whatever is in the column descriptor. But if set to false and column descriptor is set to PREFETCH_ON_OPEN => true, then the column descriptor setting prevails. The new key I added is meant to override both of those settings if set, and have no effect if not set. Notice that it has no default. So it will only have an effect if you set it.
{quote}
{code:java}
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
{code}
The file has moved to hbase-mapreduce module.
{quote}
 

 

All my patch changes are against HBase 1.4.0 . Should I rebase them to the master branch? I was afraid that would be too different from the version I am working with, but I can rebase if required.

Cheers.

 


was (Author: saadmufti):
{quote}You put the above in Release Note field.
{quote}
Removed.
{quote}
{code:java}
+  public static final String PREFETCH_BLOCKS_ON_OPEN_OVERRIDE_KEY =
+    "hbase.rs.prefetchblocksonopen.override";
{code}
There is config, "hbase.rs.prefetchblocksonopen" already. Have you tried overriding value for this config ?
{quote}
hbase.rs.prefetchblocksonopen if set to true, overrides whatever is in the column descriptor. But if set to false and column descriptor is set to PREFETCH_ON_OPEN => true, then the column descriptor setting prevails. The new key I added is meant to override both of those settings if set, and have no effect if not set. Notice that it has no default. So it will only have an effect if you set it.
{quote}
{code:java}
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
{code}
The file has moved to hbase-mapreduce module.
{quote}
 

 

All my patch changes are against HBase 1.4.0 . Should I rebase them to the master branch? I was afraid that would be too different from the version I am working, but I can rebase if required.

Cheers.

 

> Proposed Performance Enhancements For TableSnapshotInputFomat
> -------------------------------------------------------------
>
>                 Key: HBASE-20218
>                 URL: https://issues.apache.org/jira/browse/HBASE-20218
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>    Affects Versions: 1.4.0
>         Environment: HBase 1.4.0 running in AWS EMR 5.12.0 with the HBase rootdir set to a folder in S3
>  
>            Reporter: Saad Mufti
>            Priority: Minor
>         Attachments: patch.txt
>
>
> I have been testing a few Spark jobs we have at my company which work off of TableSnapshotInputFormat to read directly from the filesystem snapshots created on another EMR/Hbase cluster and stored in S3. During performance testing I found various small changes which would greatly enhance peformance. Right now we are running our jobs linked with a patched version of HBase 1.4.0 in which I made these changes, and I am hoping to submit my patch for review and eventual acceptance into the main codebase.
>  
> The list of changes are :
>  
> 1. a flag to control whether the snapshot restore uses a UUID based random temp dir in the specified restore directory. We use the flag to turn this off so that we can benefit from a AWS S3 specific bucket partitioning scheme we have provisioned. The way S3 partitioning works, you have to give a fixed path prefix and a pattern of files after that, and AWS can then partition on the paths after the fixed prefix into different resources to get more parallelization. We were advised by AWS that we could only get this good partitioning behavior if we didn't have that rancom directory in there.
>  
> 2. a flag to turn off the  code that tries to compute locality information for the splits. This is useless when dealing with S3 since the files are not on the cluster so there is no use in computing locality; and worse yet, it uses a single thread in the driver to iterate over all the files in the restored snapshot. For a very large table this was taking hours and hours iterating through S3 objects just to list them (about 360,000 of them for the our specific table).
>  
> 3. a flag to override the column family schema setting to prefetch regions on open. This was causing the main executor thread on which a Spark task was running, which was trying to read through HFile's for its scan, compete for a lock on the underlying EMRFS stream object with prefetch threads trying to read the same file, so most tasks in the Spark stage would finish but the last few would linger half an hour or more competing with the prefetch threads alternately for a lock on an EMRFS stream object. This is the only change that had to be outside the mapreduce package as it directly affects the prefetch behavior in CacheConfig.java
>  
> 4. a flag to turn off maintenance of Scan metrics. this was also causing a major slowdown, getting rid of this sped things up 4-5 times. What I observed in the thread dumps was that every call to update scan metrics was trying to get some HBase counter object and deep underneath was trying to access some Java resource bundle, and throwing an exception that it wasn't found. The exception was never visible at the application level and was swallowed underneath but whatever it was doing was causing a major slowdown. So we use this flag to avoid collecting those metrics because we never used them
>  
> I am polishing my patch a bit more and hopefully will attach it tomorrow. One caveat, I tried but struggled with how to write any useful unit/component tests for these as these are invisible behaviors that do not affect the final result at all. And I am not that familiar with the HBase testing standards, so for now I am looking for guidance on what to tests. 
>  
> Would appreciate any feedback plus guidance on writing tests, provided of course there is interest in incorporating my patch into the main codebase.
>  
> Cheers.
>  
> ----Saad
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)