[jira] [Commented] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors

["Dave Latham (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648808#comment-14648808 ] 

Dave Latham commented on HBASE-12219:
-------------------------------------

Sorry I missed this JIRA at the time, but I have a couple of concerns if I'm understanding this change correctly.  I'd like to check to see if I am.  It looks like the effect is to back out the directory modtime caching and instead have the master just maintain a persistent in memory cache.

Previous to this change it was safe to have any processes on the cluster use FSTableDescriptors to read or write table descriptors.  Updates would be atomic and consistent, immediately available to all readers.  However, the cost of that was a HDFS NN operation on every table descriptor read to prove it had not changed.  Since in practice it appears that only the master process ever updates an existing table descriptor, it should be safe to have the master skip the directory modtime checks and proactively update its cached copy.  (hbck can also create table descriptors for orphaned tables but hopefully those don't happen to tables the master has already cached).

This change makes the master descriptor reads faster but imposes the constraint that only the active master should update table descriptors - any other writers would cause the master cache to become stale.  It also means that no other processes should use the cache the same way as the master could change the data and cause stale caches.  Assuming this is the case, I think we'd be better served by reflecting that in the FSTableDescriptors API and javadoc.  For example, currently most constructors and usages now default to keeping a persistent cache as well as allowing updates which sets a bad example for new uses.  There are also no warnings in the javadoc about the new contract.  Possibly better would be to make the default constructor be read only and have persistent caching disabled.  Then another constructor for the master allowing both writes and persistent caching.

This change also seems to remove all table descriptor caching from the region servers (the old directory modtime caching is gone and the new caching is disabled for region servers).  Thanks to HBASE-8778 reloading from the FS each time is cheaper than it used to be, but this change still increases the cost from 1 NN operation (check directory modtime) to 2 NN + 3 DN operations (find current file, get its block locations, open block, read close block).  This slows things down a bit again for mass assignments/balances on huge tables.  It seems better for the region servers to retain the directory modtime caching, but simply skip the modtime check when running inside the master.

Does that understanding of this change sound correct - or did I botch it?  Sorry I missed it at the time.  If that sounds right, a follow up JIRA may be good, and if I see our table assignments slower from this and no one else gets to it I can try to put up the changes.

> Cache more efficiently getAll() and get() in FSTableDescriptors
> ---------------------------------------------------------------
>
>                 Key: HBASE-12219
>                 URL: https://issues.apache.org/jira/browse/HBASE-12219
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.94.24, 0.99.1, 0.98.6.1
>            Reporter: Esteban Gutierrez
>            Assignee: Esteban Gutierrez
>              Labels: scalability
>             Fix For: 2.0.0, 0.98.8, 0.99.2
>
>         Attachments: HBASE-12219-0.94.patch, HBASE-12219-0.98.patch, HBASE-12219-0.98.v1.addendum.patch, HBASE-12219-0.98.v1.patch, HBASE-12219-0.99.addendum.patch, HBASE-12219-0.99.patch, HBASE-12219-0.99.v1.patch, HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png
>
>
> Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and  FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor()



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)