[jira] [Commented] (HADOOP-18617) Make IOStatisticsStore and binding APIs public for use beyond our code

["Viraj Jasani (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684040#comment-17684040 ] 

Viraj Jasani commented on HADOOP-18617:
---------------------------------------

{quote}Ideally we should actually move the IOStatisticsStore interface into org.apache.hadoop.fs.statistics and the builder to match -but we can't do that without causing trauma elsewhere (google gcs).

Strategy there: Add a new interface IOStatisticsCollector in .impl which is then implemented by IOStatisticsStore, and a new builder API which forwards to IOStatisticsStoreBuilder.
{quote}
If we do this
 * Create new interface IOStatisticsCollector in .impl
 * Move interface IOStatisticsStore to org.apache.hadoop.fs.statistics
 * Make interface IOStatisticsStore implement IOStatisticsCollector (which now belongs to .impl)

We would essentially let an interface at *_xyz_* package implement another interface from *_xyz.impl_* package. I wonder if this makes the structure look a bit tricky.

> Make IOStatisticsStore and binding APIs public for use beyond our code
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-18617
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18617
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 3.3.5
>            Reporter: Steve Loughran
>            Priority: Major
>
> it's really useful to be able to collect iostats in things other than the FS classes -we do it in the S3A and manifest committers.
> But external code -such as the spark committers can't use the methods in {{org.apache.hadoop.fs.statistics.impl))
> Proposed
> Make some classes/interfaces public
> * IOStatisticsBinding
> * IOStatisticsStore
> * IOStatisticsStoreBuilder
> Ideally we should actually move the IOStatisticsStore interface into org.apache.hadoop.fs.statistics and the builder to match -but we can't do that without causing trauma elsewhere (google gcs).
> Strategy there: Add a new interface IOStatisticsCollector in .impl which is then implemented by IOStatisticsStore, and a new builder API which forwards to IOStatisticsStoreBuilder.
> Side issue: we don't make any use of the "clever, elegant functional" bit of DynamicIOStatisticsBuilder/DynamicIOStatistics, where every counter is mapped to a function which is then invoked to get at the atomic longs. It's used in IOStatisticsStoreImpl, but only with AtomicLong and MeanStatistic instances. If we just move to simple maps we will save on lambda-expressions and on lookup overhead. The original intent was something like coda hale metrics where we could add dynamic lookup to other bits of instrumentation; in practise we measure durations and build counts/min/max.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org