You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Stephen O'Donnell <so...@cloudera.com.INVALID> on 2020/02/04 18:40:43 UTC
Alternative Decommission Monitor Implementation
Hi All,
In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new
decommission monitor to trunk, which is disabled by default. The new
implementation hopes to be an improvement over the original monitor, but
without running it on a real cluster it is hard to know for sure.
I would like to ask if anyone has tried to use this new monitor in a
production cluster, and if so did you find any problems or did it work as
expected etc? I would be very interested in any feedback from anyone who
has tried to use it.
Thanks,
Stephen.
Re: Alternative Decommission Monitor Implementation
Posted by Stephen O'Donnell <so...@cloudera.com.INVALID>.
Akira,
We would be very interested in any feedback on the new monitor
implementation, as we have not yet been able to test it on a large
production cluster.
Particularly:
1. Does it work correctly at scale?
2. Does it appear to place less load on the NN (less locking, and it should
do less checking so less CPU used overall)?
3. Does it appear to be faster?
For speed, I would not expect massive gains, as the actual replication is
still done the same way. However due to how it randomises the blocks
scheduled for replication, it should do a better job of spreading the load
across the DNs. Therefore if many nodes are being decommissioned at once,
it might perform a bit better.
I am happy to help with any questions or help with fixing any issues in
this area.
Thanks,
Stephen.
On Mon, Sep 28, 2020 at 11:26 AM Akira Ajisaka <aa...@apache.org> wrote:
> Hi Wei-Chiu and Stephen,
>
> We have upgraded to HDFS 3.3.0 and experimenting the new decommission
> monitor.
> I'd like to feedback to the community in the near future.
>
> Thanks,
> Akira
>
> On Thu, Feb 6, 2020 at 2:45 PM Akira Ajisaka <aa...@apache.org> wrote:
> >
> > Yes, I'm interested in the new decommission monitor.
> > We're planning to adopt this after upgrading to HDFS 3.3.0.
> >
> > Thanks,
> > Akira
> >
> > On Wed, Feb 5, 2020 at 3:57 AM Wei-Chiu Chuang <we...@cloudera.com>
> wrote:
> >>
> >> @Akira Ajisaka you said you'd be interested, right? Are you planning to
> adopt this feature?
> >>
> >> On Tue, Feb 4, 2020 at 10:41 AM Stephen O'Donnell <
> sodonnell@cloudera.com.invalid> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new
> >>> decommission monitor to trunk, which is disabled by default. The new
> >>> implementation hopes to be an improvement over the original monitor,
> but
> >>> without running it on a real cluster it is hard to know for sure.
> >>>
> >>> I would like to ask if anyone has tried to use this new monitor in a
> >>> production cluster, and if so did you find any problems or did it work
> as
> >>> expected etc? I would be very interested in any feedback from anyone
> who
> >>> has tried to use it.
> >>>
> >>> Thanks,
> >>>
> >>> Stephen.
>
Re: Alternative Decommission Monitor Implementation
Posted by Akira Ajisaka <aa...@apache.org>.
Hi Wei-Chiu and Stephen,
We have upgraded to HDFS 3.3.0 and experimenting the new decommission monitor.
I'd like to feedback to the community in the near future.
Thanks,
Akira
On Thu, Feb 6, 2020 at 2:45 PM Akira Ajisaka <aa...@apache.org> wrote:
>
> Yes, I'm interested in the new decommission monitor.
> We're planning to adopt this after upgrading to HDFS 3.3.0.
>
> Thanks,
> Akira
>
> On Wed, Feb 5, 2020 at 3:57 AM Wei-Chiu Chuang <we...@cloudera.com> wrote:
>>
>> @Akira Ajisaka you said you'd be interested, right? Are you planning to adopt this feature?
>>
>> On Tue, Feb 4, 2020 at 10:41 AM Stephen O'Donnell <so...@cloudera.com.invalid> wrote:
>>>
>>> Hi All,
>>>
>>> In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new
>>> decommission monitor to trunk, which is disabled by default. The new
>>> implementation hopes to be an improvement over the original monitor, but
>>> without running it on a real cluster it is hard to know for sure.
>>>
>>> I would like to ask if anyone has tried to use this new monitor in a
>>> production cluster, and if so did you find any problems or did it work as
>>> expected etc? I would be very interested in any feedback from anyone who
>>> has tried to use it.
>>>
>>> Thanks,
>>>
>>> Stephen.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: Alternative Decommission Monitor Implementation
Posted by Akira Ajisaka <aa...@apache.org>.
Yes, I'm interested in the new decommission monitor.
We're planning to adopt this after upgrading to HDFS 3.3.0.
Thanks,
Akira
On Wed, Feb 5, 2020 at 3:57 AM Wei-Chiu Chuang <we...@cloudera.com> wrote:
> @Akira Ajisaka <aa...@apache.org> you said you'd be interested, right?
> Are you planning to adopt this feature?
>
> On Tue, Feb 4, 2020 at 10:41 AM Stephen O'Donnell
> <so...@cloudera.com.invalid> wrote:
>
>> Hi All,
>>
>> In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new
>> decommission monitor to trunk, which is disabled by default. The new
>> implementation hopes to be an improvement over the original monitor, but
>> without running it on a real cluster it is hard to know for sure.
>>
>> I would like to ask if anyone has tried to use this new monitor in a
>> production cluster, and if so did you find any problems or did it work as
>> expected etc? I would be very interested in any feedback from anyone who
>> has tried to use it.
>>
>> Thanks,
>>
>> Stephen.
>>
>
Re: Alternative Decommission Monitor Implementation
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
@Akira Ajisaka <aa...@apache.org> you said you'd be interested, right?
Are you planning to adopt this feature?
On Tue, Feb 4, 2020 at 10:41 AM Stephen O'Donnell
<so...@cloudera.com.invalid> wrote:
> Hi All,
>
> In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new
> decommission monitor to trunk, which is disabled by default. The new
> implementation hopes to be an improvement over the original monitor, but
> without running it on a real cluster it is hard to know for sure.
>
> I would like to ask if anyone has tried to use this new monitor in a
> production cluster, and if so did you find any problems or did it work as
> expected etc? I would be very interested in any feedback from anyone who
> has tried to use it.
>
> Thanks,
>
> Stephen.
>