You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Daniel Haviv <da...@gmail.com> on 2015/01/17 18:17:02 UTC
Edits log apply performance
Hi,
After restarting the namenode we discovered that there was no checkpoint for quite a while.
We are waiting for all the changes to be applied to the fsimage, but it seems like it will take hours.
Is there something we can do to expedite the process? Increases parallelism? Something at all?
Thanks,
Daniel
Re: Edits log apply performance
Posted by Daniel Haviv <da...@gmail.com>.
Thought so...
Thank you for the info and assistance.
BR,
Daniel
> On 19 בינו׳ 2015, at 19:48, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> Hi Daniel,
>
> Unfortunately, there likely isn't anything you can do to speed this up. The process of applying edits will be bound by the available read throughput on the edits stream and the CPU needs for processing each transaction. This is inherently a single-threaded process in the current implementation.
>
> You can check progress by accessing the NameNode web UI and looking at the Startup Progress tab. This won't tell you exactly when it will be done, but the stats on the page will give you a sense for how quickly it is making progress.
>
> For the future, you might want to consider deploying monitoring to check the time of the last checkpoint, and alert if it goes beyond some threshold. This might give you a chance to react and fix it before the next NameNode restart.
>
> There are a few improvements planned in this area for increasing performance and reducing the chance of running without a recent checkpoint. If you're interested, you can watch the following issues in jira.
>
> https://issues.apache.org/jira/browse/HDFS-4923
>
> https://issues.apache.org/jira/browse/HDFS-6353
>
> https://issues.apache.org/jira/browse/HDFS-7609
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>> On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
>> Hi,
>> After restarting the namenode we discovered that there was no checkpoint for quite a while.
>> We are waiting for all the changes to be applied to the fsimage, but it seems like it will take hours.
>>
>> Is there something we can do to expedite the process? Increases parallelism? Something at all?
>>
>> Thanks,
>> Daniel
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Daniel Haviv <da...@gmail.com>.
Thought so...
Thank you for the info and assistance.
BR,
Daniel
> On 19 בינו׳ 2015, at 19:48, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> Hi Daniel,
>
> Unfortunately, there likely isn't anything you can do to speed this up. The process of applying edits will be bound by the available read throughput on the edits stream and the CPU needs for processing each transaction. This is inherently a single-threaded process in the current implementation.
>
> You can check progress by accessing the NameNode web UI and looking at the Startup Progress tab. This won't tell you exactly when it will be done, but the stats on the page will give you a sense for how quickly it is making progress.
>
> For the future, you might want to consider deploying monitoring to check the time of the last checkpoint, and alert if it goes beyond some threshold. This might give you a chance to react and fix it before the next NameNode restart.
>
> There are a few improvements planned in this area for increasing performance and reducing the chance of running without a recent checkpoint. If you're interested, you can watch the following issues in jira.
>
> https://issues.apache.org/jira/browse/HDFS-4923
>
> https://issues.apache.org/jira/browse/HDFS-6353
>
> https://issues.apache.org/jira/browse/HDFS-7609
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>> On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
>> Hi,
>> After restarting the namenode we discovered that there was no checkpoint for quite a while.
>> We are waiting for all the changes to be applied to the fsimage, but it seems like it will take hours.
>>
>> Is there something we can do to expedite the process? Increases parallelism? Something at all?
>>
>> Thanks,
>> Daniel
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Daniel Haviv <da...@gmail.com>.
Thought so...
Thank you for the info and assistance.
BR,
Daniel
> On 19 בינו׳ 2015, at 19:48, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> Hi Daniel,
>
> Unfortunately, there likely isn't anything you can do to speed this up. The process of applying edits will be bound by the available read throughput on the edits stream and the CPU needs for processing each transaction. This is inherently a single-threaded process in the current implementation.
>
> You can check progress by accessing the NameNode web UI and looking at the Startup Progress tab. This won't tell you exactly when it will be done, but the stats on the page will give you a sense for how quickly it is making progress.
>
> For the future, you might want to consider deploying monitoring to check the time of the last checkpoint, and alert if it goes beyond some threshold. This might give you a chance to react and fix it before the next NameNode restart.
>
> There are a few improvements planned in this area for increasing performance and reducing the chance of running without a recent checkpoint. If you're interested, you can watch the following issues in jira.
>
> https://issues.apache.org/jira/browse/HDFS-4923
>
> https://issues.apache.org/jira/browse/HDFS-6353
>
> https://issues.apache.org/jira/browse/HDFS-7609
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>> On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
>> Hi,
>> After restarting the namenode we discovered that there was no checkpoint for quite a while.
>> We are waiting for all the changes to be applied to the fsimage, but it seems like it will take hours.
>>
>> Is there something we can do to expedite the process? Increases parallelism? Something at all?
>>
>> Thanks,
>> Daniel
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Daniel Haviv <da...@gmail.com>.
Thought so...
Thank you for the info and assistance.
BR,
Daniel
> On 19 בינו׳ 2015, at 19:48, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> Hi Daniel,
>
> Unfortunately, there likely isn't anything you can do to speed this up. The process of applying edits will be bound by the available read throughput on the edits stream and the CPU needs for processing each transaction. This is inherently a single-threaded process in the current implementation.
>
> You can check progress by accessing the NameNode web UI and looking at the Startup Progress tab. This won't tell you exactly when it will be done, but the stats on the page will give you a sense for how quickly it is making progress.
>
> For the future, you might want to consider deploying monitoring to check the time of the last checkpoint, and alert if it goes beyond some threshold. This might give you a chance to react and fix it before the next NameNode restart.
>
> There are a few improvements planned in this area for increasing performance and reducing the chance of running without a recent checkpoint. If you're interested, you can watch the following issues in jira.
>
> https://issues.apache.org/jira/browse/HDFS-4923
>
> https://issues.apache.org/jira/browse/HDFS-6353
>
> https://issues.apache.org/jira/browse/HDFS-7609
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>> On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
>> Hi,
>> After restarting the namenode we discovered that there was no checkpoint for quite a while.
>> We are waiting for all the changes to be applied to the fsimage, but it seems like it will take hours.
>>
>> Is there something we can do to expedite the process? Increases parallelism? Something at all?
>>
>> Thanks,
>> Daniel
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Daniel,
Unfortunately, there likely isn't anything you can do to speed this up.
The process of applying edits will be bound by the available read
throughput on the edits stream and the CPU needs for processing each
transaction. This is inherently a single-threaded process in the current
implementation.
You can check progress by accessing the NameNode web UI and looking at the
Startup Progress tab. This won't tell you exactly when it will be done,
but the stats on the page will give you a sense for how quickly it is
making progress.
For the future, you might want to consider deploying monitoring to check
the time of the last checkpoint, and alert if it goes beyond some
threshold. This might give you a chance to react and fix it before the
next NameNode restart.
There are a few improvements planned in this area for increasing
performance and reducing the chance of running without a recent
checkpoint. If you're interested, you can watch the following issues in
jira.
https://issues.apache.org/jira/browse/HDFS-4923
https://issues.apache.org/jira/browse/HDFS-6353
https://issues.apache.org/jira/browse/HDFS-7609
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
> Hi,
> After restarting the namenode we discovered that there was no checkpoint
> for quite a while.
> We are waiting for all the changes to be applied to the fsimage, but it
> seems like it will take hours.
>
> Is there something we can do to expedite the process? Increases
> parallelism? Something at all?
>
> Thanks,
> Daniel
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Daniel,
Unfortunately, there likely isn't anything you can do to speed this up.
The process of applying edits will be bound by the available read
throughput on the edits stream and the CPU needs for processing each
transaction. This is inherently a single-threaded process in the current
implementation.
You can check progress by accessing the NameNode web UI and looking at the
Startup Progress tab. This won't tell you exactly when it will be done,
but the stats on the page will give you a sense for how quickly it is
making progress.
For the future, you might want to consider deploying monitoring to check
the time of the last checkpoint, and alert if it goes beyond some
threshold. This might give you a chance to react and fix it before the
next NameNode restart.
There are a few improvements planned in this area for increasing
performance and reducing the chance of running without a recent
checkpoint. If you're interested, you can watch the following issues in
jira.
https://issues.apache.org/jira/browse/HDFS-4923
https://issues.apache.org/jira/browse/HDFS-6353
https://issues.apache.org/jira/browse/HDFS-7609
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
> Hi,
> After restarting the namenode we discovered that there was no checkpoint
> for quite a while.
> We are waiting for all the changes to be applied to the fsimage, but it
> seems like it will take hours.
>
> Is there something we can do to expedite the process? Increases
> parallelism? Something at all?
>
> Thanks,
> Daniel
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Daniel,
Unfortunately, there likely isn't anything you can do to speed this up.
The process of applying edits will be bound by the available read
throughput on the edits stream and the CPU needs for processing each
transaction. This is inherently a single-threaded process in the current
implementation.
You can check progress by accessing the NameNode web UI and looking at the
Startup Progress tab. This won't tell you exactly when it will be done,
but the stats on the page will give you a sense for how quickly it is
making progress.
For the future, you might want to consider deploying monitoring to check
the time of the last checkpoint, and alert if it goes beyond some
threshold. This might give you a chance to react and fix it before the
next NameNode restart.
There are a few improvements planned in this area for increasing
performance and reducing the chance of running without a recent
checkpoint. If you're interested, you can watch the following issues in
jira.
https://issues.apache.org/jira/browse/HDFS-4923
https://issues.apache.org/jira/browse/HDFS-6353
https://issues.apache.org/jira/browse/HDFS-7609
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
> Hi,
> After restarting the namenode we discovered that there was no checkpoint
> for quite a while.
> We are waiting for all the changes to be applied to the fsimage, but it
> seems like it will take hours.
>
> Is there something we can do to expedite the process? Increases
> parallelism? Something at all?
>
> Thanks,
> Daniel
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Edits log apply performance
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Daniel,
Unfortunately, there likely isn't anything you can do to speed this up.
The process of applying edits will be bound by the available read
throughput on the edits stream and the CPU needs for processing each
transaction. This is inherently a single-threaded process in the current
implementation.
You can check progress by accessing the NameNode web UI and looking at the
Startup Progress tab. This won't tell you exactly when it will be done,
but the stats on the page will give you a sense for how quickly it is
making progress.
For the future, you might want to consider deploying monitoring to check
the time of the last checkpoint, and alert if it goes beyond some
threshold. This might give you a chance to react and fix it before the
next NameNode restart.
There are a few improvements planned in this area for increasing
performance and reducing the chance of running without a recent
checkpoint. If you're interested, you can watch the following issues in
jira.
https://issues.apache.org/jira/browse/HDFS-4923
https://issues.apache.org/jira/browse/HDFS-6353
https://issues.apache.org/jira/browse/HDFS-7609
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <da...@gmail.com> wrote:
> Hi,
> After restarting the namenode we discovered that there was no checkpoint
> for quite a while.
> We are waiting for all the changes to be applied to the fsimage, but it
> seems like it will take hours.
>
> Is there something we can do to expedite the process? Increases
> parallelism? Something at all?
>
> Thanks,
> Daniel
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.