[jira] [Created] (HADOOP-17469) IOStatistics Phase II

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-17469:
---------------------------------------

             Summary: IOStatistics Phase II
                 Key: HADOOP-17469
                 URL: https://issues.apache.org/jira/browse/HADOOP-17469
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs, fs/azure, fs/s3
    Affects Versions: 3.3.1
            Reporter: Steve Loughran


Continue IOStatistics development with goals of

* Easy adoption in applications
* better instrumentation in hadoop codebase (distcp?)
* more stats in abfs and s3a connectors

A key has to be a thread level context for statistics so that app code doesn't have to explicitly ask for the stats for each worker thread. Instead 

filesystem components update the context stats as well as thread stats (when?) and then apps can pick up.

* need to manage performance by minimising inefficient lookups, lock acquisition etc on what should be memory-only ops (read()), (write()),
* and for duration tracking, cut down on calls to System.currentTime() so that only 1 should be made per operation, 
* need to propagate the context into worker threads




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org