You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Abhishek Modi <mo...@uber.com.INVALID> on 2020/07/27 19:54:30 UTC
[DISCUSS] Adding Metrics to Hudi Common
Hi Everyone!
I'm hoping to have a discussion around adding a lightweight metrics class
to Hudi Common. There are parts of Hudi Common that have large performance
implications, and I think adding metrics to these parts will help us track
Hudi's health in production and help us understand the performance
implications of changes we make.
I've opened a Jira on this topic -
https://issues.apache.org/jira/browse/HUDI-1025. This jira
specifically suggests adding HoodieWrapperFileSystem as this class has
performance implications not just for Hudi, but also for the underlying
DFS.
Looking forward to everyone's opinions on this :)
Best,
Modi
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by Shiyan Xu <xu...@gmail.com>.
yea make sense to keep module-specific metrics classes, like deltastreamer
metrics should just reside in hudi-utilities.
On Tue, Jul 28, 2020 at 9:52 AM Vinoth Chandar <vi...@apache.org> wrote:
> IMO having metrics within each module is probably more maintainable.
> the common metrics interfaces/base classes can just live in hudi-common for
> now?
>
> On Tue, Jul 28, 2020 at 9:06 AM Shiyan Xu <xu...@gmail.com>
> wrote:
>
> > +1. It would be very helpful to have more internal
> performance/cost-related
> > metrics (perhaps optionally enabled). Also it does make sense to move
> > metrics classes to common, or even to a separate module (if the scope
> gets
> > extended a lot further)
> >
> > On Tue, Jul 28, 2020 at 8:43 AM vbalaji@apache.org <vb...@apache.org>
> > wrote:
> >
> > > +1. Would love to see observability metrics exposed for file system
> RPC
> > > calls. This would greatly help in figuring out RPC performance and
> > > bottlenecks across varied file-systems that Hudi supports.
> > > On Tuesday, July 28, 2020, 08:24:54 AM PDT, Nishith <
> > > n3.nash29@gmail.com> wrote:
> > >
> > > +1
> > >
> > > Having the metrics flexibly in common will help in building
> observability
> > > in other modules.
> > >
> > > Thanks,
> > > Nishith
> > >
> > > > On Jul 28, 2020, at 7:28 AM, Vinoth Chandar <vi...@apache.org>
> wrote:
> > > >
> > > > +1 as well.
> > > >
> > > > Given we support many reporters now. Could you please further
> > > > improve/retain modularity.
> > > >
> > > >> On Mon, Jul 27, 2020 at 6:30 PM vino yang <ya...@gmail.com>
> > > wrote:
> > > >>
> > > >> Hi Modi,
> > > >>
> > > >> +1 for this proposal.
> > > >>
> > > >> I agree with your opinion that the metric report should not only
> > report
> > > the
> > > >> client's metrics.
> > > >>
> > > >> And we should decouple the implementation of metrics from the client
> > > module
> > > >> so that it could be developed independently.
> > > >>
> > > >> Best,
> > > >> Vino
> > > >>
> > > >> Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
> > > >>
> > > >>> Hi Everyone!
> > > >>>
> > > >>> I'm hoping to have a discussion around adding a lightweight metrics
> > > class
> > > >>> to Hudi Common. There are parts of Hudi Common that have large
> > > >> performance
> > > >>> implications, and I think adding metrics to these parts will help
> us
> > > >> track
> > > >>> Hudi's health in production and help us understand the performance
> > > >>> implications of changes we make.
> > > >>>
> > > >>> I've opened a Jira on this topic -
> > > >>> https://issues.apache.org/jira/browse/HUDI-1025. This jira
> > > >>> specifically suggests adding HoodieWrapperFileSystem as this class
> > has
> > > >>> performance implications not just for Hudi, but also for the
> > underlying
> > > >>> DFS.
> > > >>>
> > > >>> Looking forward to everyone's opinions on this :)
> > > >>>
> > > >>> Best,
> > > >>> Modi
> > > >>>
> > > >>
> >
>
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by Vinoth Chandar <vi...@apache.org>.
IMO having metrics within each module is probably more maintainable.
the common metrics interfaces/base classes can just live in hudi-common for
now?
On Tue, Jul 28, 2020 at 9:06 AM Shiyan Xu <xu...@gmail.com>
wrote:
> +1. It would be very helpful to have more internal performance/cost-related
> metrics (perhaps optionally enabled). Also it does make sense to move
> metrics classes to common, or even to a separate module (if the scope gets
> extended a lot further)
>
> On Tue, Jul 28, 2020 at 8:43 AM vbalaji@apache.org <vb...@apache.org>
> wrote:
>
> > +1. Would love to see observability metrics exposed for file system RPC
> > calls. This would greatly help in figuring out RPC performance and
> > bottlenecks across varied file-systems that Hudi supports.
> > On Tuesday, July 28, 2020, 08:24:54 AM PDT, Nishith <
> > n3.nash29@gmail.com> wrote:
> >
> > +1
> >
> > Having the metrics flexibly in common will help in building observability
> > in other modules.
> >
> > Thanks,
> > Nishith
> >
> > > On Jul 28, 2020, at 7:28 AM, Vinoth Chandar <vi...@apache.org> wrote:
> > >
> > > +1 as well.
> > >
> > > Given we support many reporters now. Could you please further
> > > improve/retain modularity.
> > >
> > >> On Mon, Jul 27, 2020 at 6:30 PM vino yang <ya...@gmail.com>
> > wrote:
> > >>
> > >> Hi Modi,
> > >>
> > >> +1 for this proposal.
> > >>
> > >> I agree with your opinion that the metric report should not only
> report
> > the
> > >> client's metrics.
> > >>
> > >> And we should decouple the implementation of metrics from the client
> > module
> > >> so that it could be developed independently.
> > >>
> > >> Best,
> > >> Vino
> > >>
> > >> Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
> > >>
> > >>> Hi Everyone!
> > >>>
> > >>> I'm hoping to have a discussion around adding a lightweight metrics
> > class
> > >>> to Hudi Common. There are parts of Hudi Common that have large
> > >> performance
> > >>> implications, and I think adding metrics to these parts will help us
> > >> track
> > >>> Hudi's health in production and help us understand the performance
> > >>> implications of changes we make.
> > >>>
> > >>> I've opened a Jira on this topic -
> > >>> https://issues.apache.org/jira/browse/HUDI-1025. This jira
> > >>> specifically suggests adding HoodieWrapperFileSystem as this class
> has
> > >>> performance implications not just for Hudi, but also for the
> underlying
> > >>> DFS.
> > >>>
> > >>> Looking forward to everyone's opinions on this :)
> > >>>
> > >>> Best,
> > >>> Modi
> > >>>
> > >>
>
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by Shiyan Xu <xu...@gmail.com>.
+1. It would be very helpful to have more internal performance/cost-related
metrics (perhaps optionally enabled). Also it does make sense to move
metrics classes to common, or even to a separate module (if the scope gets
extended a lot further)
On Tue, Jul 28, 2020 at 8:43 AM vbalaji@apache.org <vb...@apache.org>
wrote:
> +1. Would love to see observability metrics exposed for file system RPC
> calls. This would greatly help in figuring out RPC performance and
> bottlenecks across varied file-systems that Hudi supports.
> On Tuesday, July 28, 2020, 08:24:54 AM PDT, Nishith <
> n3.nash29@gmail.com> wrote:
>
> +1
>
> Having the metrics flexibly in common will help in building observability
> in other modules.
>
> Thanks,
> Nishith
>
> > On Jul 28, 2020, at 7:28 AM, Vinoth Chandar <vi...@apache.org> wrote:
> >
> > +1 as well.
> >
> > Given we support many reporters now. Could you please further
> > improve/retain modularity.
> >
> >> On Mon, Jul 27, 2020 at 6:30 PM vino yang <ya...@gmail.com>
> wrote:
> >>
> >> Hi Modi,
> >>
> >> +1 for this proposal.
> >>
> >> I agree with your opinion that the metric report should not only report
> the
> >> client's metrics.
> >>
> >> And we should decouple the implementation of metrics from the client
> module
> >> so that it could be developed independently.
> >>
> >> Best,
> >> Vino
> >>
> >> Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
> >>
> >>> Hi Everyone!
> >>>
> >>> I'm hoping to have a discussion around adding a lightweight metrics
> class
> >>> to Hudi Common. There are parts of Hudi Common that have large
> >> performance
> >>> implications, and I think adding metrics to these parts will help us
> >> track
> >>> Hudi's health in production and help us understand the performance
> >>> implications of changes we make.
> >>>
> >>> I've opened a Jira on this topic -
> >>> https://issues.apache.org/jira/browse/HUDI-1025. This jira
> >>> specifically suggests adding HoodieWrapperFileSystem as this class has
> >>> performance implications not just for Hudi, but also for the underlying
> >>> DFS.
> >>>
> >>> Looking forward to everyone's opinions on this :)
> >>>
> >>> Best,
> >>> Modi
> >>>
> >>
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by "vbalaji@apache.org" <vb...@apache.org>.
+1. Would love to see observability metrics exposed for file system RPC calls. This would greatly help in figuring out RPC performance and bottlenecks across varied file-systems that Hudi supports.
On Tuesday, July 28, 2020, 08:24:54 AM PDT, Nishith <n3...@gmail.com> wrote:
+1
Having the metrics flexibly in common will help in building observability in other modules.
Thanks,
Nishith
> On Jul 28, 2020, at 7:28 AM, Vinoth Chandar <vi...@apache.org> wrote:
>
> +1 as well.
>
> Given we support many reporters now. Could you please further
> improve/retain modularity.
>
>> On Mon, Jul 27, 2020 at 6:30 PM vino yang <ya...@gmail.com> wrote:
>>
>> Hi Modi,
>>
>> +1 for this proposal.
>>
>> I agree with your opinion that the metric report should not only report the
>> client's metrics.
>>
>> And we should decouple the implementation of metrics from the client module
>> so that it could be developed independently.
>>
>> Best,
>> Vino
>>
>> Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
>>
>>> Hi Everyone!
>>>
>>> I'm hoping to have a discussion around adding a lightweight metrics class
>>> to Hudi Common. There are parts of Hudi Common that have large
>> performance
>>> implications, and I think adding metrics to these parts will help us
>> track
>>> Hudi's health in production and help us understand the performance
>>> implications of changes we make.
>>>
>>> I've opened a Jira on this topic -
>>> https://issues.apache.org/jira/browse/HUDI-1025. This jira
>>> specifically suggests adding HoodieWrapperFileSystem as this class has
>>> performance implications not just for Hudi, but also for the underlying
>>> DFS.
>>>
>>> Looking forward to everyone's opinions on this :)
>>>
>>> Best,
>>> Modi
>>>
>>
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by Nishith <n3...@gmail.com>.
+1
Having the metrics flexibly in common will help in building observability in other modules.
Thanks,
Nishith
> On Jul 28, 2020, at 7:28 AM, Vinoth Chandar <vi...@apache.org> wrote:
>
> +1 as well.
>
> Given we support many reporters now. Could you please further
> improve/retain modularity.
>
>> On Mon, Jul 27, 2020 at 6:30 PM vino yang <ya...@gmail.com> wrote:
>>
>> Hi Modi,
>>
>> +1 for this proposal.
>>
>> I agree with your opinion that the metric report should not only report the
>> client's metrics.
>>
>> And we should decouple the implementation of metrics from the client module
>> so that it could be developed independently.
>>
>> Best,
>> Vino
>>
>> Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
>>
>>> Hi Everyone!
>>>
>>> I'm hoping to have a discussion around adding a lightweight metrics class
>>> to Hudi Common. There are parts of Hudi Common that have large
>> performance
>>> implications, and I think adding metrics to these parts will help us
>> track
>>> Hudi's health in production and help us understand the performance
>>> implications of changes we make.
>>>
>>> I've opened a Jira on this topic -
>>> https://issues.apache.org/jira/browse/HUDI-1025. This jira
>>> specifically suggests adding HoodieWrapperFileSystem as this class has
>>> performance implications not just for Hudi, but also for the underlying
>>> DFS.
>>>
>>> Looking forward to everyone's opinions on this :)
>>>
>>> Best,
>>> Modi
>>>
>>
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by Vinoth Chandar <vi...@apache.org>.
+1 as well.
Given we support many reporters now. Could you please further
improve/retain modularity.
On Mon, Jul 27, 2020 at 6:30 PM vino yang <ya...@gmail.com> wrote:
> Hi Modi,
>
> +1 for this proposal.
>
> I agree with your opinion that the metric report should not only report the
> client's metrics.
>
> And we should decouple the implementation of metrics from the client module
> so that it could be developed independently.
>
> Best,
> Vino
>
> Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
>
> > Hi Everyone!
> >
> > I'm hoping to have a discussion around adding a lightweight metrics class
> > to Hudi Common. There are parts of Hudi Common that have large
> performance
> > implications, and I think adding metrics to these parts will help us
> track
> > Hudi's health in production and help us understand the performance
> > implications of changes we make.
> >
> > I've opened a Jira on this topic -
> > https://issues.apache.org/jira/browse/HUDI-1025. This jira
> > specifically suggests adding HoodieWrapperFileSystem as this class has
> > performance implications not just for Hudi, but also for the underlying
> > DFS.
> >
> > Looking forward to everyone's opinions on this :)
> >
> > Best,
> > Modi
> >
>
Re: [DISCUSS] Adding Metrics to Hudi Common
Posted by vino yang <ya...@gmail.com>.
Hi Modi,
+1 for this proposal.
I agree with your opinion that the metric report should not only report the
client's metrics.
And we should decouple the implementation of metrics from the client module
so that it could be developed independently.
Best,
Vino
Abhishek Modi <mo...@uber.com.invalid> 于2020年7月28日周二 上午4:17写道:
> Hi Everyone!
>
> I'm hoping to have a discussion around adding a lightweight metrics class
> to Hudi Common. There are parts of Hudi Common that have large performance
> implications, and I think adding metrics to these parts will help us track
> Hudi's health in production and help us understand the performance
> implications of changes we make.
>
> I've opened a Jira on this topic -
> https://issues.apache.org/jira/browse/HUDI-1025. This jira
> specifically suggests adding HoodieWrapperFileSystem as this class has
> performance implications not just for Hudi, but also for the underlying
> DFS.
>
> Looking forward to everyone's opinions on this :)
>
> Best,
> Modi
>