You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Ayush Saxena <ay...@gmail.com> on 2020/02/04 08:20:49 UTC
Restrict Frequency of BlockReport To Namenode startup and failover
Hi All,
Me and Surendra have been lately trying to minimise the impact of Block Reports on Namenode in huge cluster. We observed in a huge cluster, about 10k datanodes, the periodic block reports impact the Namenode performance adversely.
We have been thinking to restrict the block reports to be triggered only during Namenode startup or in case of failover and eliminate the periodic block report.
The main purpose of block report is to get a corrupt blocks recognised, so as a follow up we can maintain a service at datanode to run periodically to check if the block size in memory is same as that reported to namenode, and the datanode can alarm the namenode in case of any suspect,(We still need to plan this.)
At the datanode side, a datanode can send a BlockReport or restore its actual frequency in case during the configured time period, the Datanode got shutdown or lost connection with the namenode, say if the datanode was supposed to send BR at 2100 hrs, if during the last 6 hrs there has been any failover or loss of connection between the namenode and datanode, it will trigger BR normally, else shall skip sending the BR
Let us know thoughts/challenges/improvements in this.
-Ayush
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Ayush Saxena <ay...@gmail.com>.
Thanx Everyone!!!
Just to conclude the thread.
Have created HDFS-15162 to track this.
-Ayush
> On 09-Feb-2020, at 5:01 PM, Ayush Saxena <ay...@gmail.com> wrote:
>
> Hi Stephen,
> We are trying this on 3.1.1
> We aren’t upgrading from 2.x, we are trying to increase the cluster size to go beyond 10K datanodes.
> In the process, we analysed that block reports from these many DN’s are quite bothersome.
> There are plenty of reasons why block reports bothers performance, the major being namenode holding the lock for these many datanodes, as you mentioned.
> HDFS-14657 may improve the situation a bit(I didn’t follow it) but our point is rather than improving the impact, we can completely get rid of them in most of the cases.
>
> Why to unnecessarily have load of processing Block Reports, if it isn’t doing anything good.
>
> So, just wanted to know, if people are aware of any cases where eliminating regular BR’s can be a problem, which we might have missed.
>
> Let me know if you possess hard feelings for the change or doubt something.
>
> -Ayush
>
>>> On 07-Feb-2020, at 4:03 PM, Stephen O'Donnell <so...@cloudera.com> wrote:
>>>
>>
>> Are you seeing this problem on the 3.x branch, and if so, did the problem exist before you upgraded to 3.x? I am wondering if the situation is better or worse since moving to 3.x.
>>
>> Also, do you believe the issue is driven by the namenode holding its lock for too long while it processes each block report, blocking other threads?
>>
>> There was an interesting proposal in https://issues.apache.org/jira/browse/HDFS-14657 to allow the NN lock to be dropped and retaken periodically while processing FBRs, but it has not progressed recently. I wonder if that would help here?
>>
>> Thanks,
>>
>> Stephen.
>>
>>> On Fri, Feb 7, 2020 at 6:58 AM Surendra Singh Lilhore <su...@apache.org> wrote:
>>> Thanks Wei-Chiu,
>>>
>>> I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
>>> bugs in IBR, I feel we should fix such bug in IBR. Adding one new
>>> functionality to prevent bug in other is not good.
>>>
>>> I also thing, DN should send BR in failure and process start scenario only.
>>>
>>> -Surendra
>>>
>>> On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>>
>>> > Hi Wei-Chiu,
>>> > Thanx for the response.
>>> > Yes, We are talking about the FBR only.
>>> > Increasing the frequency limits the problem, but doesn’t seems to be
>>> > solving it. With increasing cluster size, the frequency needs to be
>>> > increased, and we cannot increase it indefinitely, as in some case FBR is
>>> > needed.
>>> > One such case is Namenode failover, In case of failover the namenode marks
>>> > all the storages as Stale, it would correct them only once FBR comes, Any
>>> > overreplicated blocks won’t be deleted until the storages are in stale
>>> > state.
>>> >
>>> > Regarding the IBR error, the block is set Completed post IBR, when the
>>> > client claimed value and IBR values matches, so if there is a discrepancy
>>> > here, it would alarm out there itself.
>>> >
>>> > If it passes over this spot, so the FBR would also be sending the same
>>> > values from memory, it doesn’t check from the actual disk.
>>> > DirectoryScanner would be checking if the in memory data is same as that
>>> > on the disk.
>>> > Other scenario where FBR could be needed is to counter a split brain
>>> > scenario, but with QJM’s that is unlikely to happen.
>>> >
>>> > In case of any connection losses during the interval, we tend to send the
>>> > BR, so should be safe here.
>>> >
>>> > Anyway if a client gets hold of a invalid block, it will too report to the
>>> > Namenode.
>>> >
>>> > Other we cannot think as such, where not sending FBR can cause any issue.
>>> >
>>> > Let us know your thoughts on this..
>>> >
>>> > -Ayush
>>> >
>>> > >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
>>> > wrote:
>>> > >> Hey Ayush,
>>> > >>
>>> > >> Thanks a lot for your proposal.
>>> > >>
>>> > >> Do you mean the Full Block Report that is sent out every 6 hours per
>>> > >> DataNode?
>>> > >> Someone told me they reduced the frequency of FBR to 24 hours and it
>>> > seems
>>> > >> okay.
>>> > >>
>>> > >> One of the purposes of FBR was to prevent bugs in incremental block
>>> > report
>>> > >> implementation. In other words, it's a fail-safe mechanism. Any bugs in
>>> > >> IBRs get corrected after a FBR that refreshes the state of blocks at
>>> > >> NameNode. At least, that's my understanding of FBRs in its early days.
>>> > >>
>>> > >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
>>> > wrote:
>>> > >>
>>> > >> Hi All,
>>> > >> Me and Surendra have been lately trying to minimise the impact of Block
>>> > >> Reports on Namenode in huge cluster. We observed in a huge cluster,
>>> > about
>>> > >> 10k datanodes, the periodic block reports impact the Namenode
>>> > performance
>>> > >> adversely.
>>> > >> We have been thinking to restrict the block reports to be triggered only
>>> > >> during Namenode startup or in case of failover and eliminate the
>>> > periodic
>>> > >> block report.
>>> > >> The main purpose of block report is to get a corrupt blocks recognised,
>>> > so
>>> > >> as a follow up we can maintain a service at datanode to run
>>> > periodically to
>>> > >> check if the block size in memory is same as that reported to namenode,
>>> > and
>>> > >> the datanode can alarm the namenode in case of any suspect,(We still
>>> > need
>>> > >> to plan this.)
>>> > >>
>>> > >> At the datanode side, a datanode can send a BlockReport or restore its
>>> > >> actual frequency in case during the configured time period, the Datanode
>>> > >> got shutdown or lost connection with the namenode, say if the datanode
>>> > was
>>> > >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
>>> > >> any failover or loss of connection between the namenode and datanode, it
>>> > >> will trigger BR normally, else shall skip sending the BR
>>> > >>
>>> > >> Let us know thoughts/challenges/improvements in this.
>>> > >>
>>> > >> -Ayush
>>> > >>
>>> > >>
>>> > >>
>>> > >> ---------------------------------------------------------------------
>>> > >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> > >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>> >
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Ayush Saxena <ay...@gmail.com>.
Thanx Everyone!!!
Just to conclude the thread.
Have created HDFS-15162 to track this.
-Ayush
> On 09-Feb-2020, at 5:01 PM, Ayush Saxena <ay...@gmail.com> wrote:
>
> Hi Stephen,
> We are trying this on 3.1.1
> We aren’t upgrading from 2.x, we are trying to increase the cluster size to go beyond 10K datanodes.
> In the process, we analysed that block reports from these many DN’s are quite bothersome.
> There are plenty of reasons why block reports bothers performance, the major being namenode holding the lock for these many datanodes, as you mentioned.
> HDFS-14657 may improve the situation a bit(I didn’t follow it) but our point is rather than improving the impact, we can completely get rid of them in most of the cases.
>
> Why to unnecessarily have load of processing Block Reports, if it isn’t doing anything good.
>
> So, just wanted to know, if people are aware of any cases where eliminating regular BR’s can be a problem, which we might have missed.
>
> Let me know if you possess hard feelings for the change or doubt something.
>
> -Ayush
>
>>> On 07-Feb-2020, at 4:03 PM, Stephen O'Donnell <so...@cloudera.com> wrote:
>>>
>>
>> Are you seeing this problem on the 3.x branch, and if so, did the problem exist before you upgraded to 3.x? I am wondering if the situation is better or worse since moving to 3.x.
>>
>> Also, do you believe the issue is driven by the namenode holding its lock for too long while it processes each block report, blocking other threads?
>>
>> There was an interesting proposal in https://issues.apache.org/jira/browse/HDFS-14657 to allow the NN lock to be dropped and retaken periodically while processing FBRs, but it has not progressed recently. I wonder if that would help here?
>>
>> Thanks,
>>
>> Stephen.
>>
>>> On Fri, Feb 7, 2020 at 6:58 AM Surendra Singh Lilhore <su...@apache.org> wrote:
>>> Thanks Wei-Chiu,
>>>
>>> I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
>>> bugs in IBR, I feel we should fix such bug in IBR. Adding one new
>>> functionality to prevent bug in other is not good.
>>>
>>> I also thing, DN should send BR in failure and process start scenario only.
>>>
>>> -Surendra
>>>
>>> On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>>
>>> > Hi Wei-Chiu,
>>> > Thanx for the response.
>>> > Yes, We are talking about the FBR only.
>>> > Increasing the frequency limits the problem, but doesn’t seems to be
>>> > solving it. With increasing cluster size, the frequency needs to be
>>> > increased, and we cannot increase it indefinitely, as in some case FBR is
>>> > needed.
>>> > One such case is Namenode failover, In case of failover the namenode marks
>>> > all the storages as Stale, it would correct them only once FBR comes, Any
>>> > overreplicated blocks won’t be deleted until the storages are in stale
>>> > state.
>>> >
>>> > Regarding the IBR error, the block is set Completed post IBR, when the
>>> > client claimed value and IBR values matches, so if there is a discrepancy
>>> > here, it would alarm out there itself.
>>> >
>>> > If it passes over this spot, so the FBR would also be sending the same
>>> > values from memory, it doesn’t check from the actual disk.
>>> > DirectoryScanner would be checking if the in memory data is same as that
>>> > on the disk.
>>> > Other scenario where FBR could be needed is to counter a split brain
>>> > scenario, but with QJM’s that is unlikely to happen.
>>> >
>>> > In case of any connection losses during the interval, we tend to send the
>>> > BR, so should be safe here.
>>> >
>>> > Anyway if a client gets hold of a invalid block, it will too report to the
>>> > Namenode.
>>> >
>>> > Other we cannot think as such, where not sending FBR can cause any issue.
>>> >
>>> > Let us know your thoughts on this..
>>> >
>>> > -Ayush
>>> >
>>> > >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
>>> > wrote:
>>> > >> Hey Ayush,
>>> > >>
>>> > >> Thanks a lot for your proposal.
>>> > >>
>>> > >> Do you mean the Full Block Report that is sent out every 6 hours per
>>> > >> DataNode?
>>> > >> Someone told me they reduced the frequency of FBR to 24 hours and it
>>> > seems
>>> > >> okay.
>>> > >>
>>> > >> One of the purposes of FBR was to prevent bugs in incremental block
>>> > report
>>> > >> implementation. In other words, it's a fail-safe mechanism. Any bugs in
>>> > >> IBRs get corrected after a FBR that refreshes the state of blocks at
>>> > >> NameNode. At least, that's my understanding of FBRs in its early days.
>>> > >>
>>> > >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
>>> > wrote:
>>> > >>
>>> > >> Hi All,
>>> > >> Me and Surendra have been lately trying to minimise the impact of Block
>>> > >> Reports on Namenode in huge cluster. We observed in a huge cluster,
>>> > about
>>> > >> 10k datanodes, the periodic block reports impact the Namenode
>>> > performance
>>> > >> adversely.
>>> > >> We have been thinking to restrict the block reports to be triggered only
>>> > >> during Namenode startup or in case of failover and eliminate the
>>> > periodic
>>> > >> block report.
>>> > >> The main purpose of block report is to get a corrupt blocks recognised,
>>> > so
>>> > >> as a follow up we can maintain a service at datanode to run
>>> > periodically to
>>> > >> check if the block size in memory is same as that reported to namenode,
>>> > and
>>> > >> the datanode can alarm the namenode in case of any suspect,(We still
>>> > need
>>> > >> to plan this.)
>>> > >>
>>> > >> At the datanode side, a datanode can send a BlockReport or restore its
>>> > >> actual frequency in case during the configured time period, the Datanode
>>> > >> got shutdown or lost connection with the namenode, say if the datanode
>>> > was
>>> > >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
>>> > >> any failover or loss of connection between the namenode and datanode, it
>>> > >> will trigger BR normally, else shall skip sending the BR
>>> > >>
>>> > >> Let us know thoughts/challenges/improvements in this.
>>> > >>
>>> > >> -Ayush
>>> > >>
>>> > >>
>>> > >>
>>> > >> ---------------------------------------------------------------------
>>> > >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> > >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>> >
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Stephen,
We are trying this on 3.1.1
We aren’t upgrading from 2.x, we are trying to increase the cluster size to go beyond 10K datanodes.
In the process, we analysed that block reports from these many DN’s are quite bothersome.
There are plenty of reasons why block reports bothers performance, the major being namenode holding the lock for these many datanodes, as you mentioned.
HDFS-14657 may improve the situation a bit(I didn’t follow it) but our point is rather than improving the impact, we can completely get rid of them in most of the cases.
Why to unnecessarily have load of processing Block Reports, if it isn’t doing anything good.
So, just wanted to know, if people are aware of any cases where eliminating regular BR’s can be a problem, which we might have missed.
Let me know if you possess hard feelings for the change or doubt something.
-Ayush
> On 07-Feb-2020, at 4:03 PM, Stephen O'Donnell <so...@cloudera.com> wrote:
>
>
> Are you seeing this problem on the 3.x branch, and if so, did the problem exist before you upgraded to 3.x? I am wondering if the situation is better or worse since moving to 3.x.
>
> Also, do you believe the issue is driven by the namenode holding its lock for too long while it processes each block report, blocking other threads?
>
> There was an interesting proposal in https://issues.apache.org/jira/browse/HDFS-14657 to allow the NN lock to be dropped and retaken periodically while processing FBRs, but it has not progressed recently. I wonder if that would help here?
>
> Thanks,
>
> Stephen.
>
>> On Fri, Feb 7, 2020 at 6:58 AM Surendra Singh Lilhore <su...@apache.org> wrote:
>> Thanks Wei-Chiu,
>>
>> I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
>> bugs in IBR, I feel we should fix such bug in IBR. Adding one new
>> functionality to prevent bug in other is not good.
>>
>> I also thing, DN should send BR in failure and process start scenario only.
>>
>> -Surendra
>>
>> On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > Hi Wei-Chiu,
>> > Thanx for the response.
>> > Yes, We are talking about the FBR only.
>> > Increasing the frequency limits the problem, but doesn’t seems to be
>> > solving it. With increasing cluster size, the frequency needs to be
>> > increased, and we cannot increase it indefinitely, as in some case FBR is
>> > needed.
>> > One such case is Namenode failover, In case of failover the namenode marks
>> > all the storages as Stale, it would correct them only once FBR comes, Any
>> > overreplicated blocks won’t be deleted until the storages are in stale
>> > state.
>> >
>> > Regarding the IBR error, the block is set Completed post IBR, when the
>> > client claimed value and IBR values matches, so if there is a discrepancy
>> > here, it would alarm out there itself.
>> >
>> > If it passes over this spot, so the FBR would also be sending the same
>> > values from memory, it doesn’t check from the actual disk.
>> > DirectoryScanner would be checking if the in memory data is same as that
>> > on the disk.
>> > Other scenario where FBR could be needed is to counter a split brain
>> > scenario, but with QJM’s that is unlikely to happen.
>> >
>> > In case of any connection losses during the interval, we tend to send the
>> > BR, so should be safe here.
>> >
>> > Anyway if a client gets hold of a invalid block, it will too report to the
>> > Namenode.
>> >
>> > Other we cannot think as such, where not sending FBR can cause any issue.
>> >
>> > Let us know your thoughts on this..
>> >
>> > -Ayush
>> >
>> > >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
>> > wrote:
>> > >> Hey Ayush,
>> > >>
>> > >> Thanks a lot for your proposal.
>> > >>
>> > >> Do you mean the Full Block Report that is sent out every 6 hours per
>> > >> DataNode?
>> > >> Someone told me they reduced the frequency of FBR to 24 hours and it
>> > seems
>> > >> okay.
>> > >>
>> > >> One of the purposes of FBR was to prevent bugs in incremental block
>> > report
>> > >> implementation. In other words, it's a fail-safe mechanism. Any bugs in
>> > >> IBRs get corrected after a FBR that refreshes the state of blocks at
>> > >> NameNode. At least, that's my understanding of FBRs in its early days.
>> > >>
>> > >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
>> > wrote:
>> > >>
>> > >> Hi All,
>> > >> Me and Surendra have been lately trying to minimise the impact of Block
>> > >> Reports on Namenode in huge cluster. We observed in a huge cluster,
>> > about
>> > >> 10k datanodes, the periodic block reports impact the Namenode
>> > performance
>> > >> adversely.
>> > >> We have been thinking to restrict the block reports to be triggered only
>> > >> during Namenode startup or in case of failover and eliminate the
>> > periodic
>> > >> block report.
>> > >> The main purpose of block report is to get a corrupt blocks recognised,
>> > so
>> > >> as a follow up we can maintain a service at datanode to run
>> > periodically to
>> > >> check if the block size in memory is same as that reported to namenode,
>> > and
>> > >> the datanode can alarm the namenode in case of any suspect,(We still
>> > need
>> > >> to plan this.)
>> > >>
>> > >> At the datanode side, a datanode can send a BlockReport or restore its
>> > >> actual frequency in case during the configured time period, the Datanode
>> > >> got shutdown or lost connection with the namenode, say if the datanode
>> > was
>> > >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
>> > >> any failover or loss of connection between the namenode and datanode, it
>> > >> will trigger BR normally, else shall skip sending the BR
>> > >>
>> > >> Let us know thoughts/challenges/improvements in this.
>> > >>
>> > >> -Ayush
>> > >>
>> > >>
>> > >>
>> > >> ---------------------------------------------------------------------
>> > >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> >
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Stephen,
We are trying this on 3.1.1
We aren’t upgrading from 2.x, we are trying to increase the cluster size to go beyond 10K datanodes.
In the process, we analysed that block reports from these many DN’s are quite bothersome.
There are plenty of reasons why block reports bothers performance, the major being namenode holding the lock for these many datanodes, as you mentioned.
HDFS-14657 may improve the situation a bit(I didn’t follow it) but our point is rather than improving the impact, we can completely get rid of them in most of the cases.
Why to unnecessarily have load of processing Block Reports, if it isn’t doing anything good.
So, just wanted to know, if people are aware of any cases where eliminating regular BR’s can be a problem, which we might have missed.
Let me know if you possess hard feelings for the change or doubt something.
-Ayush
> On 07-Feb-2020, at 4:03 PM, Stephen O'Donnell <so...@cloudera.com> wrote:
>
>
> Are you seeing this problem on the 3.x branch, and if so, did the problem exist before you upgraded to 3.x? I am wondering if the situation is better or worse since moving to 3.x.
>
> Also, do you believe the issue is driven by the namenode holding its lock for too long while it processes each block report, blocking other threads?
>
> There was an interesting proposal in https://issues.apache.org/jira/browse/HDFS-14657 to allow the NN lock to be dropped and retaken periodically while processing FBRs, but it has not progressed recently. I wonder if that would help here?
>
> Thanks,
>
> Stephen.
>
>> On Fri, Feb 7, 2020 at 6:58 AM Surendra Singh Lilhore <su...@apache.org> wrote:
>> Thanks Wei-Chiu,
>>
>> I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
>> bugs in IBR, I feel we should fix such bug in IBR. Adding one new
>> functionality to prevent bug in other is not good.
>>
>> I also thing, DN should send BR in failure and process start scenario only.
>>
>> -Surendra
>>
>> On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> > Hi Wei-Chiu,
>> > Thanx for the response.
>> > Yes, We are talking about the FBR only.
>> > Increasing the frequency limits the problem, but doesn’t seems to be
>> > solving it. With increasing cluster size, the frequency needs to be
>> > increased, and we cannot increase it indefinitely, as in some case FBR is
>> > needed.
>> > One such case is Namenode failover, In case of failover the namenode marks
>> > all the storages as Stale, it would correct them only once FBR comes, Any
>> > overreplicated blocks won’t be deleted until the storages are in stale
>> > state.
>> >
>> > Regarding the IBR error, the block is set Completed post IBR, when the
>> > client claimed value and IBR values matches, so if there is a discrepancy
>> > here, it would alarm out there itself.
>> >
>> > If it passes over this spot, so the FBR would also be sending the same
>> > values from memory, it doesn’t check from the actual disk.
>> > DirectoryScanner would be checking if the in memory data is same as that
>> > on the disk.
>> > Other scenario where FBR could be needed is to counter a split brain
>> > scenario, but with QJM’s that is unlikely to happen.
>> >
>> > In case of any connection losses during the interval, we tend to send the
>> > BR, so should be safe here.
>> >
>> > Anyway if a client gets hold of a invalid block, it will too report to the
>> > Namenode.
>> >
>> > Other we cannot think as such, where not sending FBR can cause any issue.
>> >
>> > Let us know your thoughts on this..
>> >
>> > -Ayush
>> >
>> > >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
>> > wrote:
>> > >> Hey Ayush,
>> > >>
>> > >> Thanks a lot for your proposal.
>> > >>
>> > >> Do you mean the Full Block Report that is sent out every 6 hours per
>> > >> DataNode?
>> > >> Someone told me they reduced the frequency of FBR to 24 hours and it
>> > seems
>> > >> okay.
>> > >>
>> > >> One of the purposes of FBR was to prevent bugs in incremental block
>> > report
>> > >> implementation. In other words, it's a fail-safe mechanism. Any bugs in
>> > >> IBRs get corrected after a FBR that refreshes the state of blocks at
>> > >> NameNode. At least, that's my understanding of FBRs in its early days.
>> > >>
>> > >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
>> > wrote:
>> > >>
>> > >> Hi All,
>> > >> Me and Surendra have been lately trying to minimise the impact of Block
>> > >> Reports on Namenode in huge cluster. We observed in a huge cluster,
>> > about
>> > >> 10k datanodes, the periodic block reports impact the Namenode
>> > performance
>> > >> adversely.
>> > >> We have been thinking to restrict the block reports to be triggered only
>> > >> during Namenode startup or in case of failover and eliminate the
>> > periodic
>> > >> block report.
>> > >> The main purpose of block report is to get a corrupt blocks recognised,
>> > so
>> > >> as a follow up we can maintain a service at datanode to run
>> > periodically to
>> > >> check if the block size in memory is same as that reported to namenode,
>> > and
>> > >> the datanode can alarm the namenode in case of any suspect,(We still
>> > need
>> > >> to plan this.)
>> > >>
>> > >> At the datanode side, a datanode can send a BlockReport or restore its
>> > >> actual frequency in case during the configured time period, the Datanode
>> > >> got shutdown or lost connection with the namenode, say if the datanode
>> > was
>> > >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
>> > >> any failover or loss of connection between the namenode and datanode, it
>> > >> will trigger BR normally, else shall skip sending the BR
>> > >>
>> > >> Let us know thoughts/challenges/improvements in this.
>> > >>
>> > >> -Ayush
>> > >>
>> > >>
>> > >>
>> > >> ---------------------------------------------------------------------
>> > >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> > >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>> >
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Stephen O'Donnell <so...@cloudera.com.INVALID>.
Are you seeing this problem on the 3.x branch, and if so, did the problem
exist before you upgraded to 3.x? I am wondering if the situation is better
or worse since moving to 3.x.
Also, do you believe the issue is driven by the namenode holding its lock
for too long while it processes each block report, blocking other threads?
There was an interesting proposal in
https://issues.apache.org/jira/browse/HDFS-14657 to allow the NN lock to be
dropped and retaken periodically while processing FBRs, but it has not
progressed recently. I wonder if that would help here?
Thanks,
Stephen.
On Fri, Feb 7, 2020 at 6:58 AM Surendra Singh Lilhore <
surendralilhore@apache.org> wrote:
> Thanks Wei-Chiu,
>
> I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
> bugs in IBR, I feel we should fix such bug in IBR. Adding one new
> functionality to prevent bug in other is not good.
>
> I also thing, DN should send BR in failure and process start scenario only.
>
> -Surendra
>
> On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hi Wei-Chiu,
> > Thanx for the response.
> > Yes, We are talking about the FBR only.
> > Increasing the frequency limits the problem, but doesn’t seems to be
> > solving it. With increasing cluster size, the frequency needs to be
> > increased, and we cannot increase it indefinitely, as in some case FBR is
> > needed.
> > One such case is Namenode failover, In case of failover the namenode
> marks
> > all the storages as Stale, it would correct them only once FBR comes, Any
> > overreplicated blocks won’t be deleted until the storages are in stale
> > state.
> >
> > Regarding the IBR error, the block is set Completed post IBR, when the
> > client claimed value and IBR values matches, so if there is a discrepancy
> > here, it would alarm out there itself.
> >
> > If it passes over this spot, so the FBR would also be sending the same
> > values from memory, it doesn’t check from the actual disk.
> > DirectoryScanner would be checking if the in memory data is same as that
> > on the disk.
> > Other scenario where FBR could be needed is to counter a split brain
> > scenario, but with QJM’s that is unlikely to happen.
> >
> > In case of any connection losses during the interval, we tend to send the
> > BR, so should be safe here.
> >
> > Anyway if a client gets hold of a invalid block, it will too report to
> the
> > Namenode.
> >
> > Other we cannot think as such, where not sending FBR can cause any issue.
> >
> > Let us know your thoughts on this..
> >
> > -Ayush
> >
> > >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> > >> Hey Ayush,
> > >>
> > >> Thanks a lot for your proposal.
> > >>
> > >> Do you mean the Full Block Report that is sent out every 6 hours per
> > >> DataNode?
> > >> Someone told me they reduced the frequency of FBR to 24 hours and it
> > seems
> > >> okay.
> > >>
> > >> One of the purposes of FBR was to prevent bugs in incremental block
> > report
> > >> implementation. In other words, it's a fail-safe mechanism. Any bugs
> in
> > >> IBRs get corrected after a FBR that refreshes the state of blocks at
> > >> NameNode. At least, that's my understanding of FBRs in its early days.
> > >>
> > >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
> > wrote:
> > >>
> > >> Hi All,
> > >> Me and Surendra have been lately trying to minimise the impact of
> Block
> > >> Reports on Namenode in huge cluster. We observed in a huge cluster,
> > about
> > >> 10k datanodes, the periodic block reports impact the Namenode
> > performance
> > >> adversely.
> > >> We have been thinking to restrict the block reports to be triggered
> only
> > >> during Namenode startup or in case of failover and eliminate the
> > periodic
> > >> block report.
> > >> The main purpose of block report is to get a corrupt blocks
> recognised,
> > so
> > >> as a follow up we can maintain a service at datanode to run
> > periodically to
> > >> check if the block size in memory is same as that reported to
> namenode,
> > and
> > >> the datanode can alarm the namenode in case of any suspect,(We still
> > need
> > >> to plan this.)
> > >>
> > >> At the datanode side, a datanode can send a BlockReport or restore its
> > >> actual frequency in case during the configured time period, the
> Datanode
> > >> got shutdown or lost connection with the namenode, say if the datanode
> > was
> > >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has
> been
> > >> any failover or loss of connection between the namenode and datanode,
> it
> > >> will trigger BR normally, else shall skip sending the BR
> > >>
> > >> Let us know thoughts/challenges/improvements in this.
> > >>
> > >> -Ayush
> > >>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Surendra Singh Lilhore <su...@apache.org>.
Thanks Wei-Chiu,
I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
bugs in IBR, I feel we should fix such bug in IBR. Adding one new
functionality to prevent bug in other is not good.
I also thing, DN should send BR in failure and process start scenario only.
-Surendra
On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Wei-Chiu,
> Thanx for the response.
> Yes, We are talking about the FBR only.
> Increasing the frequency limits the problem, but doesn’t seems to be
> solving it. With increasing cluster size, the frequency needs to be
> increased, and we cannot increase it indefinitely, as in some case FBR is
> needed.
> One such case is Namenode failover, In case of failover the namenode marks
> all the storages as Stale, it would correct them only once FBR comes, Any
> overreplicated blocks won’t be deleted until the storages are in stale
> state.
>
> Regarding the IBR error, the block is set Completed post IBR, when the
> client claimed value and IBR values matches, so if there is a discrepancy
> here, it would alarm out there itself.
>
> If it passes over this spot, so the FBR would also be sending the same
> values from memory, it doesn’t check from the actual disk.
> DirectoryScanner would be checking if the in memory data is same as that
> on the disk.
> Other scenario where FBR could be needed is to counter a split brain
> scenario, but with QJM’s that is unlikely to happen.
>
> In case of any connection losses during the interval, we tend to send the
> BR, so should be safe here.
>
> Anyway if a client gets hold of a invalid block, it will too report to the
> Namenode.
>
> Other we cannot think as such, where not sending FBR can cause any issue.
>
> Let us know your thoughts on this..
>
> -Ayush
>
> >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >> Hey Ayush,
> >>
> >> Thanks a lot for your proposal.
> >>
> >> Do you mean the Full Block Report that is sent out every 6 hours per
> >> DataNode?
> >> Someone told me they reduced the frequency of FBR to 24 hours and it
> seems
> >> okay.
> >>
> >> One of the purposes of FBR was to prevent bugs in incremental block
> report
> >> implementation. In other words, it's a fail-safe mechanism. Any bugs in
> >> IBRs get corrected after a FBR that refreshes the state of blocks at
> >> NameNode. At least, that's my understanding of FBRs in its early days.
> >>
> >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
> wrote:
> >>
> >> Hi All,
> >> Me and Surendra have been lately trying to minimise the impact of Block
> >> Reports on Namenode in huge cluster. We observed in a huge cluster,
> about
> >> 10k datanodes, the periodic block reports impact the Namenode
> performance
> >> adversely.
> >> We have been thinking to restrict the block reports to be triggered only
> >> during Namenode startup or in case of failover and eliminate the
> periodic
> >> block report.
> >> The main purpose of block report is to get a corrupt blocks recognised,
> so
> >> as a follow up we can maintain a service at datanode to run
> periodically to
> >> check if the block size in memory is same as that reported to namenode,
> and
> >> the datanode can alarm the namenode in case of any suspect,(We still
> need
> >> to plan this.)
> >>
> >> At the datanode side, a datanode can send a BlockReport or restore its
> >> actual frequency in case during the configured time period, the Datanode
> >> got shutdown or lost connection with the namenode, say if the datanode
> was
> >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
> >> any failover or loss of connection between the namenode and datanode, it
> >> will trigger BR normally, else shall skip sending the BR
> >>
> >> Let us know thoughts/challenges/improvements in this.
> >>
> >> -Ayush
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Surendra Singh Lilhore <su...@apache.org>.
Thanks Wei-Chiu,
I feel now IBR is more stable in branch 3.x. If BR is just added to prevent
bugs in IBR, I feel we should fix such bug in IBR. Adding one new
functionality to prevent bug in other is not good.
I also thing, DN should send BR in failure and process start scenario only.
-Surendra
On Fri, Feb 7, 2020 at 10:52 AM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Wei-Chiu,
> Thanx for the response.
> Yes, We are talking about the FBR only.
> Increasing the frequency limits the problem, but doesn’t seems to be
> solving it. With increasing cluster size, the frequency needs to be
> increased, and we cannot increase it indefinitely, as in some case FBR is
> needed.
> One such case is Namenode failover, In case of failover the namenode marks
> all the storages as Stale, it would correct them only once FBR comes, Any
> overreplicated blocks won’t be deleted until the storages are in stale
> state.
>
> Regarding the IBR error, the block is set Completed post IBR, when the
> client claimed value and IBR values matches, so if there is a discrepancy
> here, it would alarm out there itself.
>
> If it passes over this spot, so the FBR would also be sending the same
> values from memory, it doesn’t check from the actual disk.
> DirectoryScanner would be checking if the in memory data is same as that
> on the disk.
> Other scenario where FBR could be needed is to counter a split brain
> scenario, but with QJM’s that is unlikely to happen.
>
> In case of any connection losses during the interval, we tend to send the
> BR, so should be safe here.
>
> Anyway if a client gets hold of a invalid block, it will too report to the
> Namenode.
>
> Other we cannot think as such, where not sending FBR can cause any issue.
>
> Let us know your thoughts on this..
>
> -Ayush
>
> >>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org>
> wrote:
> >> Hey Ayush,
> >>
> >> Thanks a lot for your proposal.
> >>
> >> Do you mean the Full Block Report that is sent out every 6 hours per
> >> DataNode?
> >> Someone told me they reduced the frequency of FBR to 24 hours and it
> seems
> >> okay.
> >>
> >> One of the purposes of FBR was to prevent bugs in incremental block
> report
> >> implementation. In other words, it's a fail-safe mechanism. Any bugs in
> >> IBRs get corrected after a FBR that refreshes the state of blocks at
> >> NameNode. At least, that's my understanding of FBRs in its early days.
> >>
> >> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com>
> wrote:
> >>
> >> Hi All,
> >> Me and Surendra have been lately trying to minimise the impact of Block
> >> Reports on Namenode in huge cluster. We observed in a huge cluster,
> about
> >> 10k datanodes, the periodic block reports impact the Namenode
> performance
> >> adversely.
> >> We have been thinking to restrict the block reports to be triggered only
> >> during Namenode startup or in case of failover and eliminate the
> periodic
> >> block report.
> >> The main purpose of block report is to get a corrupt blocks recognised,
> so
> >> as a follow up we can maintain a service at datanode to run
> periodically to
> >> check if the block size in memory is same as that reported to namenode,
> and
> >> the datanode can alarm the namenode in case of any suspect,(We still
> need
> >> to plan this.)
> >>
> >> At the datanode side, a datanode can send a BlockReport or restore its
> >> actual frequency in case during the configured time period, the Datanode
> >> got shutdown or lost connection with the namenode, say if the datanode
> was
> >> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
> >> any failover or loss of connection between the namenode and datanode, it
> >> will trigger BR normally, else shall skip sending the BR
> >>
> >> Let us know thoughts/challenges/improvements in this.
> >>
> >> -Ayush
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Wei-Chiu,
Thanx for the response.
Yes, We are talking about the FBR only.
Increasing the frequency limits the problem, but doesn’t seems to be solving it. With increasing cluster size, the frequency needs to be increased, and we cannot increase it indefinitely, as in some case FBR is needed.
One such case is Namenode failover, In case of failover the namenode marks all the storages as Stale, it would correct them only once FBR comes, Any overreplicated blocks won’t be deleted until the storages are in stale state.
Regarding the IBR error, the block is set Completed post IBR, when the client claimed value and IBR values matches, so if there is a discrepancy here, it would alarm out there itself.
If it passes over this spot, so the FBR would also be sending the same values from memory, it doesn’t check from the actual disk.
DirectoryScanner would be checking if the in memory data is same as that on the disk.
Other scenario where FBR could be needed is to counter a split brain scenario, but with QJM’s that is unlikely to happen.
In case of any connection losses during the interval, we tend to send the BR, so should be safe here.
Anyway if a client gets hold of a invalid block, it will too report to the Namenode.
Other we cannot think as such, where not sending FBR can cause any issue.
Let us know your thoughts on this..
-Ayush
>>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
>> Hey Ayush,
>>
>> Thanks a lot for your proposal.
>>
>> Do you mean the Full Block Report that is sent out every 6 hours per
>> DataNode?
>> Someone told me they reduced the frequency of FBR to 24 hours and it seems
>> okay.
>>
>> One of the purposes of FBR was to prevent bugs in incremental block report
>> implementation. In other words, it's a fail-safe mechanism. Any bugs in
>> IBRs get corrected after a FBR that refreshes the state of blocks at
>> NameNode. At least, that's my understanding of FBRs in its early days.
>>
>> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> Hi All,
>> Me and Surendra have been lately trying to minimise the impact of Block
>> Reports on Namenode in huge cluster. We observed in a huge cluster, about
>> 10k datanodes, the periodic block reports impact the Namenode performance
>> adversely.
>> We have been thinking to restrict the block reports to be triggered only
>> during Namenode startup or in case of failover and eliminate the periodic
>> block report.
>> The main purpose of block report is to get a corrupt blocks recognised, so
>> as a follow up we can maintain a service at datanode to run periodically to
>> check if the block size in memory is same as that reported to namenode, and
>> the datanode can alarm the namenode in case of any suspect,(We still need
>> to plan this.)
>>
>> At the datanode side, a datanode can send a BlockReport or restore its
>> actual frequency in case during the configured time period, the Datanode
>> got shutdown or lost connection with the namenode, say if the datanode was
>> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
>> any failover or loss of connection between the namenode and datanode, it
>> will trigger BR normally, else shall skip sending the BR
>>
>> Let us know thoughts/challenges/improvements in this.
>>
>> -Ayush
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Wei-Chiu,
Thanx for the response.
Yes, We are talking about the FBR only.
Increasing the frequency limits the problem, but doesn’t seems to be solving it. With increasing cluster size, the frequency needs to be increased, and we cannot increase it indefinitely, as in some case FBR is needed.
One such case is Namenode failover, In case of failover the namenode marks all the storages as Stale, it would correct them only once FBR comes, Any overreplicated blocks won’t be deleted until the storages are in stale state.
Regarding the IBR error, the block is set Completed post IBR, when the client claimed value and IBR values matches, so if there is a discrepancy here, it would alarm out there itself.
If it passes over this spot, so the FBR would also be sending the same values from memory, it doesn’t check from the actual disk.
DirectoryScanner would be checking if the in memory data is same as that on the disk.
Other scenario where FBR could be needed is to counter a split brain scenario, but with QJM’s that is unlikely to happen.
In case of any connection losses during the interval, we tend to send the BR, so should be safe here.
Anyway if a client gets hold of a invalid block, it will too report to the Namenode.
Other we cannot think as such, where not sending FBR can cause any issue.
Let us know your thoughts on this..
-Ayush
>>> On 07-Feb-2020, at 4:12 AM, Wei-Chiu Chuang <we...@apache.org> wrote:
>> Hey Ayush,
>>
>> Thanks a lot for your proposal.
>>
>> Do you mean the Full Block Report that is sent out every 6 hours per
>> DataNode?
>> Someone told me they reduced the frequency of FBR to 24 hours and it seems
>> okay.
>>
>> One of the purposes of FBR was to prevent bugs in incremental block report
>> implementation. In other words, it's a fail-safe mechanism. Any bugs in
>> IBRs get corrected after a FBR that refreshes the state of blocks at
>> NameNode. At least, that's my understanding of FBRs in its early days.
>>
>> On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> Hi All,
>> Me and Surendra have been lately trying to minimise the impact of Block
>> Reports on Namenode in huge cluster. We observed in a huge cluster, about
>> 10k datanodes, the periodic block reports impact the Namenode performance
>> adversely.
>> We have been thinking to restrict the block reports to be triggered only
>> during Namenode startup or in case of failover and eliminate the periodic
>> block report.
>> The main purpose of block report is to get a corrupt blocks recognised, so
>> as a follow up we can maintain a service at datanode to run periodically to
>> check if the block size in memory is same as that reported to namenode, and
>> the datanode can alarm the namenode in case of any suspect,(We still need
>> to plan this.)
>>
>> At the datanode side, a datanode can send a BlockReport or restore its
>> actual frequency in case during the configured time period, the Datanode
>> got shutdown or lost connection with the namenode, say if the datanode was
>> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
>> any failover or loss of connection between the namenode and datanode, it
>> will trigger BR normally, else shall skip sending the BR
>>
>> Let us know thoughts/challenges/improvements in this.
>>
>> -Ayush
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Wei-Chiu Chuang <we...@apache.org>.
Hey Ayush,
Thanks a lot for your proposal.
Do you mean the Full Block Report that is sent out every 6 hours per
DataNode?
Someone told me they reduced the frequency of FBR to 24 hours and it seems
okay.
One of the purposes of FBR was to prevent bugs in incremental block report
implementation. In other words, it's a fail-safe mechanism. Any bugs in
IBRs get corrected after a FBR that refreshes the state of blocks at
NameNode. At least, that's my understanding of FBRs in its early days.
On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com> wrote:
> Hi All,
> Me and Surendra have been lately trying to minimise the impact of Block
> Reports on Namenode in huge cluster. We observed in a huge cluster, about
> 10k datanodes, the periodic block reports impact the Namenode performance
> adversely.
> We have been thinking to restrict the block reports to be triggered only
> during Namenode startup or in case of failover and eliminate the periodic
> block report.
> The main purpose of block report is to get a corrupt blocks recognised, so
> as a follow up we can maintain a service at datanode to run periodically to
> check if the block size in memory is same as that reported to namenode, and
> the datanode can alarm the namenode in case of any suspect,(We still need
> to plan this.)
>
> At the datanode side, a datanode can send a BlockReport or restore its
> actual frequency in case during the configured time period, the Datanode
> got shutdown or lost connection with the namenode, say if the datanode was
> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
> any failover or loss of connection between the namenode and datanode, it
> will trigger BR normally, else shall skip sending the BR
>
> Let us know thoughts/challenges/improvements in this.
>
> -Ayush
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: Restrict Frequency of BlockReport To Namenode startup and failover
Posted by Wei-Chiu Chuang <we...@apache.org>.
Hey Ayush,
Thanks a lot for your proposal.
Do you mean the Full Block Report that is sent out every 6 hours per
DataNode?
Someone told me they reduced the frequency of FBR to 24 hours and it seems
okay.
One of the purposes of FBR was to prevent bugs in incremental block report
implementation. In other words, it's a fail-safe mechanism. Any bugs in
IBRs get corrected after a FBR that refreshes the state of blocks at
NameNode. At least, that's my understanding of FBRs in its early days.
On Tue, Feb 4, 2020 at 12:21 AM Ayush Saxena <ay...@gmail.com> wrote:
> Hi All,
> Me and Surendra have been lately trying to minimise the impact of Block
> Reports on Namenode in huge cluster. We observed in a huge cluster, about
> 10k datanodes, the periodic block reports impact the Namenode performance
> adversely.
> We have been thinking to restrict the block reports to be triggered only
> during Namenode startup or in case of failover and eliminate the periodic
> block report.
> The main purpose of block report is to get a corrupt blocks recognised, so
> as a follow up we can maintain a service at datanode to run periodically to
> check if the block size in memory is same as that reported to namenode, and
> the datanode can alarm the namenode in case of any suspect,(We still need
> to plan this.)
>
> At the datanode side, a datanode can send a BlockReport or restore its
> actual frequency in case during the configured time period, the Datanode
> got shutdown or lost connection with the namenode, say if the datanode was
> supposed to send BR at 2100 hrs, if during the last 6 hrs there has been
> any failover or loss of connection between the namenode and datanode, it
> will trigger BR normally, else shall skip sending the BR
>
> Let us know thoughts/challenges/improvements in this.
>
> -Ayush
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>