[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15301094#comment-15301094 ] 

stack commented on HADOOP-12910:
--------------------------------

The patch here converts one method only? Is the intent to do all methods (along w/ the spec suggested by [~steve_l]?)

Is this issue for an AsyncFileSystem or for an async rename only? Are we targeting H3 only or is there some thought that this could get pulled back into H2?

bq. Futures are a good match for the use case where the consumer wants to kick of a multitude of async requests and wait until they are all done to make progress, but we've found that there are also compelling use cases where you want a small amount of logic and further async I/O in a completion handler, so I might recommend supporting both Future-based results as well as callback-based results.

A few of us (mainly [~Apache9]), are looking at being able to go async against hdfs. There is already a stripped down async subset of dfsclient that we are using to write our WALs done by [~Apache9] that uses way less resources while going much faster (see HBASE-14790). As Duo says, we want to push this up into HDFS, and given our good experience with this effort, we want to convert over more of our HDFS connection to be async. Parking a resource waiting on a Future to complete or keeping some list/queue of Futures which we check on a period to see if it is 'done' is much less attractive (and less performant) to our being instead notified on completion -- a callback (as [~bobhansen] suggests above in the comment repeated here).  Ideally we'd like to move our interaction with HDFS to be event-driven (ultimately backing this up all the ways into the guts of the regionserver, but that is another story)

OK if we put up suggested patch that say presumes jdk8/h3 only and instead of returning Future, returns jdk8 CompletableFuture? Chatting yesterday, we think we could consume/feed HDFS in a non-blocking way if we got back a CompletableFuture (or we could add a callback handler as a param on a method if folks preferred that?). We'd put up a sketch patch, and if amenable, we could start up a sympathetic spec doc as a subtask so code and spec arrive at the same time?

Thanks.

> Add new FileSystem API to support asynchronous method calls
> -----------------------------------------------------------
>
>                 Key: HADOOP-12910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12910
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Xiaobing Zhou
>         Attachments: HADOOP-12910-HDFS-9924.000.patch, HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a better name).  All the APIs in FutureFileSystem are the same as FileSystem except that the return type is wrapped by Future, e.g.
> {code}
>   //FileSystem
>   public boolean rename(Path src, Path dst) throws IOException;
>   //FutureFileSystem
>   public Future<Boolean> rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org