[jira] [Commented] (HADOOP-16791) ABFS: Have all external dependent module execution tracked with DurationInfo

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008747#comment-17008747 ] 

Steve Loughran commented on HADOOP-16791:
-----------------------------------------


Sounds good. We've just added some speedups to DurationInfo; we should be back porting that.

Some of my colleagues (hello Hive team) have been expressing concern to me about how long it takes object store filesystems to be created once their initialize() method starts making remote HTTP calls to multiple Service endpoints. This matters because FileSystem.get() Will create multiple instances of the same FS endpoint in parallel if the initialisation is ongoing while separate thread also call the get() method. The discussion has focused around doing async init in S3A; abfs could add that too. You've have a separate thread doing the token init etc -but then every public API call would need to potentially block awaiting that init to complete.

+[~rbalamohan] [~gabor.bota]

> ABFS: Have all external dependent module execution tracked with DurationInfo
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-16791
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16791
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>            Reporter: Sneha Vijayarajan
>            Assignee: Sneha Vijayarajan
>            Priority: Major
>             Fix For: 3.3.0
>
>
> To be able to break down the perf impact of the external module executions within ABFS Driver, add execution time computation using DurationInfo in all the relative places. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org