[jira] Commented: (HDFS-1214) hdfs client metadata cache

["Joydeep Sen Sarma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879466#action_12879466 ] 

Joydeep Sen Sarma commented on HDFS-1214:
-----------------------------------------

while a cache can be maintained on the application side - it's harder and it seems like the wrong place to implement the same. in the case of a query compiler - different compilation stages may be fetching metadata to figure out cost of query. furthermore, different queries may be compiled from the same jvm that end up requesting metadata for the same objects. 

the application can identify calls that can deal with out of date metadata. (so a separate api or an overlaid filesystem driver  or additional flags in the current api are all acceptable)

ideally the cache should be write-through (it's very common for a single jvm to be reading/writing the same fs object repeatedly).

> hdfs client metadata cache
> --------------------------
>
>                 Key: HDFS-1214
>                 URL: https://issues.apache.org/jira/browse/HDFS-1214
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs client
>            Reporter: Joydeep Sen Sarma
>
> In some applications, latency is affected by the cost of making rpc calls to namenode to fetch metadata. the most obvious case are calls to fetch file/directory status. applications like hive like to make optimizations based on file size/number etc. - and for such optimizations - 'recent' status data (as opposed to most up-to-date) is acceptable. in such cases, a cache on the DFS client that transparently caches metadata would be greatly benefit applications.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.