HUDI Metrics for hudi-common components

[Prashant Wason <pw...@uber.com.INVALID>]

Hi Team,

I am interested in adding metrics support to HoodieWrapperFileSystem. This
will help track the counts of operations and their latencies and will
provide valuable data to implement and test newer ideas (e.g. RFC 15
<https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15:+HUDI+File+Listing+and+Query+Planning+Improvements>which
is
proposing a consolidated metadata reducing the number of file system
operations).

HUDI metrics are currently implemented in module hudi-client. Modules like
hudi-utilities have hudi-client as their dependency (via pom.xml). But this
cannot be done for hudi-common as this module is itself a dependency for
hudi-client.

Hence, I feel it may be better to move the HUID metrics code to hudi-common
as most modules anyways depend on hudi-common.

What do you think about this? Any other ideas of how to approach this?

Thanks
Prashant

Re: HUDI Metrics for hudi-common components

[Vinoth Chandar <vi...@apache.org>]

Hi Prashant,

Thanks for bringing this up.  I don't know if its a good idea to move "all"
of metrics code to hudi-common, since some things are purely write
specific..

But, we can selectively move some base classes there and have the actual
metrics live across both hudi-client and hudi-common?

If you are interested, in the specific problem alone, then having just a
clear Metrics class that tracks calls from the Wrapper FileSystem and
reports it back to either query engine or
write client via a callback should suffice?

Thanks
Vinoth



On Tue, Mar 10, 2020 at 1:02 PM Prashant Wason <pw...@uber.com.invalid>
wrote:

> Hi Team,
>
> I am interested in adding metrics support to HoodieWrapperFileSystem. This
> will help track the counts of operations and their latencies and will
> provide valuable data to implement and test newer ideas (e.g. RFC 15
> <
> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15:+HUDI+File+Listing+and+Query+Planning+Improvements
> >which
> is
> proposing a consolidated metadata reducing the number of file system
> operations).
>
> HUDI metrics are currently implemented in module hudi-client. Modules like
> hudi-utilities have hudi-client as their dependency (via pom.xml). But this
> cannot be done for hudi-common as this module is itself a dependency for
> hudi-client.
>
> Hence, I feel it may be better to move the HUID metrics code to hudi-common
> as most modules anyways depend on hudi-common.
>
> What do you think about this? Any other ideas of how to approach this?
>
> Thanks
> Prashant
>