Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

I have an edits file on my namenode that is 35GB. This is quite a bit
larger than it should be (the secondary namenode wasn't running for some
time, and HBASE-9648 caused a huge number of additional edits).

The first time I tried to start the namenode, it chewed on the edits for
about 4 hours and then ran out of memory. I have increased the memory
available to the namenode (was 512MB, now 2GB), and started the process
again.

Is there any way that the edits file can be partially processed to avoid
having to re-process the same edits over and over until I can allocate
enough memory for it to be done in one shot?

How long should it take (hours? days?) to process an edits file of that
size?

Any help is appreciated!

--Tom

Re: Is there any way to partially process HDFS edits?

[Jens Scheidtmann <je...@gmail.com>]

Tom,

I would file a jira, if I were you and my Hadoop Version was recent
enough.  Should be pretty easy to reproduce.

Jens

Am Donnerstag, 26. September 2013 schrieb Tom Brown :

> They were created and deleted in quick succession. I thought that meant
> the edits for both the create and delete would be logically next to each
> other in the file allowing it to release the memory almost as soon as it
> had been allocated.
>
> In any case, after finding a VM host that could give me more RAM, I was
> able to get the namenode started. The process used 25GB at it's peak.
>
> Thanks for your help!
>
>
> On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:
>
> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a
>
>

Re: Is there any way to partially process HDFS edits?

[Jens Scheidtmann <je...@gmail.com>]

Tom,

I would file a jira, if I were you and my Hadoop Version was recent
enough.  Should be pretty easy to reproduce.

Jens

Am Donnerstag, 26. September 2013 schrieb Tom Brown :

> They were created and deleted in quick succession. I thought that meant
> the edits for both the create and delete would be logically next to each
> other in the file allowing it to release the memory almost as soon as it
> had been allocated.
>
> In any case, after finding a VM host that could give me more RAM, I was
> able to get the namenode started. The process used 25GB at it's peak.
>
> Thanks for your help!
>
>
> On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:
>
> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a
>
>

Re: Is there any way to partially process HDFS edits?

[Jens Scheidtmann <je...@gmail.com>]

Tom,

I would file a jira, if I were you and my Hadoop Version was recent
enough.  Should be pretty easy to reproduce.

Jens

Am Donnerstag, 26. September 2013 schrieb Tom Brown :

> They were created and deleted in quick succession. I thought that meant
> the edits for both the create and delete would be logically next to each
> other in the file allowing it to release the memory almost as soon as it
> had been allocated.
>
> In any case, after finding a VM host that could give me more RAM, I was
> able to get the namenode started. The process used 25GB at it's peak.
>
> Thanks for your help!
>
>
> On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:
>
> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a
>
>

Re: Is there any way to partially process HDFS edits?

[Jens Scheidtmann <je...@gmail.com>]

Tom,

I would file a jira, if I were you and my Hadoop Version was recent
enough.  Should be pretty easy to reproduce.

Jens

Am Donnerstag, 26. September 2013 schrieb Tom Brown :

> They were created and deleted in quick succession. I thought that meant
> the edits for both the create and delete would be logically next to each
> other in the file allowing it to release the memory almost as soon as it
> had been allocated.
>
> In any case, after finding a VM host that could give me more RAM, I was
> able to get the namenode started. The process used 25GB at it's peak.
>
> Thanks for your help!
>
>
> On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:
>
> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

They were created and deleted in quick succession. I thought that meant the
edits for both the create and delete would be logically next to each other
in the file allowing it to release the memory almost as soon as it had been
allocated.

In any case, after finding a VM host that could give me more RAM, I was
able to get the namenode started. The process used 25GB at it's peak.

Thanks for your help!


On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:

> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a streaming form),
> >>>> > the
> >>>> > entire edits file will never be in RAM at once.
> >>>> >
> >>>> > Is the edits file format well defined (could I break off 100MB
> chunks
> >>>> > and
> >>>> > process them individually to achieve the same result as processing
> the
> >>>> > whole
> >>>> > thing at once)?
> >>>> >
> >>>> > --Tom
> >>>> >
> >>>> >
> >>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
> >>>> > wrote:
> >>>> >>
> >>>> >> Tom! I would guess that just giving the NN JVM lots of memory
> (64Gb /
> >>>> >> 96Gb) should be the easiest way.
> >>>> >>
> >>>> >>
> >>>> >> ________________________________
> >>>> >> From: Tom Brown <to...@gmail.com>
> >>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >>>> >> Subject: Is there any way to partially process HDFS edits?
> >>>> >>
> >>>> >> I have an edits file on my namenode that is 35GB. This is quite a
> bit
> >>>> >> larger than it should be (the secondary namenode wasn't running for
> >>>> >> some
> >>>> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>>> >>
> >>>> >> The first time I tried to start the namenode, it chewed on the
> edits
> >>>> >> for
> >>>> >> about 4 hours and then ran out of memory. I have increased the
> memory
> >>>> >> available to the namenode (was 512MB, now 2GB), and started the
> >>>> >> process
> >>>> >> again.
> >>>> >>
> >>>> >> Is there any way that the edits file can be partially processed to
> >>>> >> avoid
> >>>> >> having to re-process the same edits over and over until I can
> >>>> >> allocate
> >>>> >> enough memory for it to be done in one shot?
> >>>> >>
> >>>> >> How long should it take (hours? days?) to process an edits file of
> >>>> >> that
> >>>> >> size?
> >>>> >>
> >>>> >> Any help is appreciated!
> >>>> >>
> >>>> >> --Tom
> >>>> >>
> >>>> >>
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Harsh J
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Nitin Pawar
> >
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

They were created and deleted in quick succession. I thought that meant the
edits for both the create and delete would be logically next to each other
in the file allowing it to release the memory almost as soon as it had been
allocated.

In any case, after finding a VM host that could give me more RAM, I was
able to get the namenode started. The process used 25GB at it's peak.

Thanks for your help!


On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:

> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a streaming form),
> >>>> > the
> >>>> > entire edits file will never be in RAM at once.
> >>>> >
> >>>> > Is the edits file format well defined (could I break off 100MB
> chunks
> >>>> > and
> >>>> > process them individually to achieve the same result as processing
> the
> >>>> > whole
> >>>> > thing at once)?
> >>>> >
> >>>> > --Tom
> >>>> >
> >>>> >
> >>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
> >>>> > wrote:
> >>>> >>
> >>>> >> Tom! I would guess that just giving the NN JVM lots of memory
> (64Gb /
> >>>> >> 96Gb) should be the easiest way.
> >>>> >>
> >>>> >>
> >>>> >> ________________________________
> >>>> >> From: Tom Brown <to...@gmail.com>
> >>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >>>> >> Subject: Is there any way to partially process HDFS edits?
> >>>> >>
> >>>> >> I have an edits file on my namenode that is 35GB. This is quite a
> bit
> >>>> >> larger than it should be (the secondary namenode wasn't running for
> >>>> >> some
> >>>> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>>> >>
> >>>> >> The first time I tried to start the namenode, it chewed on the
> edits
> >>>> >> for
> >>>> >> about 4 hours and then ran out of memory. I have increased the
> memory
> >>>> >> available to the namenode (was 512MB, now 2GB), and started the
> >>>> >> process
> >>>> >> again.
> >>>> >>
> >>>> >> Is there any way that the edits file can be partially processed to
> >>>> >> avoid
> >>>> >> having to re-process the same edits over and over until I can
> >>>> >> allocate
> >>>> >> enough memory for it to be done in one shot?
> >>>> >>
> >>>> >> How long should it take (hours? days?) to process an edits file of
> >>>> >> that
> >>>> >> size?
> >>>> >>
> >>>> >> Any help is appreciated!
> >>>> >>
> >>>> >> --Tom
> >>>> >>
> >>>> >>
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Harsh J
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Nitin Pawar
> >
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

They were created and deleted in quick succession. I thought that meant the
edits for both the create and delete would be logically next to each other
in the file allowing it to release the memory almost as soon as it had been
allocated.

In any case, after finding a VM host that could give me more RAM, I was
able to get the namenode started. The process used 25GB at it's peak.

Thanks for your help!


On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:

> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a streaming form),
> >>>> > the
> >>>> > entire edits file will never be in RAM at once.
> >>>> >
> >>>> > Is the edits file format well defined (could I break off 100MB
> chunks
> >>>> > and
> >>>> > process them individually to achieve the same result as processing
> the
> >>>> > whole
> >>>> > thing at once)?
> >>>> >
> >>>> > --Tom
> >>>> >
> >>>> >
> >>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
> >>>> > wrote:
> >>>> >>
> >>>> >> Tom! I would guess that just giving the NN JVM lots of memory
> (64Gb /
> >>>> >> 96Gb) should be the easiest way.
> >>>> >>
> >>>> >>
> >>>> >> ________________________________
> >>>> >> From: Tom Brown <to...@gmail.com>
> >>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >>>> >> Subject: Is there any way to partially process HDFS edits?
> >>>> >>
> >>>> >> I have an edits file on my namenode that is 35GB. This is quite a
> bit
> >>>> >> larger than it should be (the secondary namenode wasn't running for
> >>>> >> some
> >>>> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>>> >>
> >>>> >> The first time I tried to start the namenode, it chewed on the
> edits
> >>>> >> for
> >>>> >> about 4 hours and then ran out of memory. I have increased the
> memory
> >>>> >> available to the namenode (was 512MB, now 2GB), and started the
> >>>> >> process
> >>>> >> again.
> >>>> >>
> >>>> >> Is there any way that the edits file can be partially processed to
> >>>> >> avoid
> >>>> >> having to re-process the same edits over and over until I can
> >>>> >> allocate
> >>>> >> enough memory for it to be done in one shot?
> >>>> >>
> >>>> >> How long should it take (hours? days?) to process an edits file of
> >>>> >> that
> >>>> >> size?
> >>>> >>
> >>>> >> Any help is appreciated!
> >>>> >>
> >>>> >> --Tom
> >>>> >>
> >>>> >>
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Harsh J
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Nitin Pawar
> >
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

They were created and deleted in quick succession. I thought that meant the
edits for both the create and delete would be logically next to each other
in the file allowing it to release the memory almost as soon as it had been
allocated.

In any case, after finding a VM host that could give me more RAM, I was
able to get the namenode started. The process used 25GB at it's peak.

Thanks for your help!


On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <ha...@cloudera.com> wrote:

> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a streaming form),
> >>>> > the
> >>>> > entire edits file will never be in RAM at once.
> >>>> >
> >>>> > Is the edits file format well defined (could I break off 100MB
> chunks
> >>>> > and
> >>>> > process them individually to achieve the same result as processing
> the
> >>>> > whole
> >>>> > thing at once)?
> >>>> >
> >>>> > --Tom
> >>>> >
> >>>> >
> >>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
> >>>> > wrote:
> >>>> >>
> >>>> >> Tom! I would guess that just giving the NN JVM lots of memory
> (64Gb /
> >>>> >> 96Gb) should be the easiest way.
> >>>> >>
> >>>> >>
> >>>> >> ________________________________
> >>>> >> From: Tom Brown <to...@gmail.com>
> >>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >>>> >> Subject: Is there any way to partially process HDFS edits?
> >>>> >>
> >>>> >> I have an edits file on my namenode that is 35GB. This is quite a
> bit
> >>>> >> larger than it should be (the secondary namenode wasn't running for
> >>>> >> some
> >>>> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>>> >>
> >>>> >> The first time I tried to start the namenode, it chewed on the
> edits
> >>>> >> for
> >>>> >> about 4 hours and then ran out of memory. I have increased the
> memory
> >>>> >> available to the namenode (was 512MB, now 2GB), and started the
> >>>> >> process
> >>>> >> again.
> >>>> >>
> >>>> >> Is there any way that the edits file can be partially processed to
> >>>> >> avoid
> >>>> >> having to re-process the same edits over and over until I can
> >>>> >> allocate
> >>>> >> enough memory for it to be done in one shot?
> >>>> >>
> >>>> >> How long should it take (hours? days?) to process an edits file of
> >>>> >> that
> >>>> >> size?
> >>>> >>
> >>>> >> Any help is appreciated!
> >>>> >>
> >>>> >> --Tom
> >>>> >>
> >>>> >>
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Harsh J
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Nitin Pawar
> >
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Tom,

That is valuable info. When we "replay" edits, we would be creating
and then deleting those files - so memory would grow in between until
the delete events begin appearing in the edit log segment.

On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> A simple estimate puts the total number of blocks somewhere around 500,000.
> Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
> that were created and quickly deleted (about 10/sec for 6 weeks) in the
> cluster, and that activity is what is contained in the edits.
>
> Since those files don't exist (quickly created and deleted), shouldn't they
> be inconsequential to the memory requirements of the namenode as it starts
> up.
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> wrote:
>>
>> Can you share how many blocks does your cluster have? how many
>> directories? how many files?
>>
>> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
>> explains how much RAM will be used for your namenode.
>> Its pretty old by hadoop version but its a good starting point.
>>
>> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
>> NameNode memory for every 1 million blocks stored in the distributed file
>> system"
>>
>> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>>
>>
>>
>> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>>>
>>> It ran again for about 15 hours before dying again. I'm seeing what extra
>>> RAM resources we can throw at this VM (maybe up to 32GB), but until then I'm
>>> trying to figure out if I'm hitting some strange bug.
>>>
>>> When the edits were originally made (over the course of 6 weeks), the
>>> namenode only had 512MB and was able to contain the filesystem completely in
>>> memory. I don't understand why it's running out of memory. If 512MB was
>>> enough while the edits were first made, shouldn't it be enough to process
>>> them again?
>>>
>>> --Tom
>>>
>>>
>>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>> Hi Tom,
>>>>
>>>> The edits are processed sequentially, and aren't all held in memory.
>>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>>> it could resume only with remaining work if interrupted. Normally this
>>>> is not a problem in deployments given that SNN or SBN runs for
>>>> checkpointing the images and keeping the edits collection small
>>>> periodically.
>>>>
>>>> If your NameNode is running out of memory _applying_ the edits, then
>>>> the cause is not the edits but a growing namespace. You most-likely
>>>> have more files now than before, and thats going to take up permanent
>>>> memory from the NameNode heap size.
>>>>
>>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>>> > total
>>>> > (though could be expanded somewhat-- it's a VM).
>>>> >
>>>> > Though if each edit is processed sequentially (in a streaming form),
>>>> > the
>>>> > entire edits file will never be in RAM at once.
>>>> >
>>>> > Is the edits file format well defined (could I break off 100MB chunks
>>>> > and
>>>> > process them individually to achieve the same result as processing the
>>>> > whole
>>>> > thing at once)?
>>>> >
>>>> > --Tom
>>>> >
>>>> >
>>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>>> > wrote:
>>>> >>
>>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>>> >> 96Gb) should be the easiest way.
>>>> >>
>>>> >>
>>>> >> ________________________________
>>>> >> From: Tom Brown <to...@gmail.com>
>>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>>> >> Subject: Is there any way to partially process HDFS edits?
>>>> >>
>>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>>> >> larger than it should be (the secondary namenode wasn't running for
>>>> >> some
>>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>>> >>
>>>> >> The first time I tried to start the namenode, it chewed on the edits
>>>> >> for
>>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>>> >> process
>>>> >> again.
>>>> >>
>>>> >> Is there any way that the edits file can be partially processed to
>>>> >> avoid
>>>> >> having to re-process the same edits over and over until I can
>>>> >> allocate
>>>> >> enough memory for it to be done in one shot?
>>>> >>
>>>> >> How long should it take (hours? days?) to process an edits file of
>>>> >> that
>>>> >> size?
>>>> >>
>>>> >> Any help is appreciated!
>>>> >>
>>>> >> --Tom
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Tom,

That is valuable info. When we "replay" edits, we would be creating
and then deleting those files - so memory would grow in between until
the delete events begin appearing in the edit log segment.

On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> A simple estimate puts the total number of blocks somewhere around 500,000.
> Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
> that were created and quickly deleted (about 10/sec for 6 weeks) in the
> cluster, and that activity is what is contained in the edits.
>
> Since those files don't exist (quickly created and deleted), shouldn't they
> be inconsequential to the memory requirements of the namenode as it starts
> up.
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> wrote:
>>
>> Can you share how many blocks does your cluster have? how many
>> directories? how many files?
>>
>> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
>> explains how much RAM will be used for your namenode.
>> Its pretty old by hadoop version but its a good starting point.
>>
>> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
>> NameNode memory for every 1 million blocks stored in the distributed file
>> system"
>>
>> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>>
>>
>>
>> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>>>
>>> It ran again for about 15 hours before dying again. I'm seeing what extra
>>> RAM resources we can throw at this VM (maybe up to 32GB), but until then I'm
>>> trying to figure out if I'm hitting some strange bug.
>>>
>>> When the edits were originally made (over the course of 6 weeks), the
>>> namenode only had 512MB and was able to contain the filesystem completely in
>>> memory. I don't understand why it's running out of memory. If 512MB was
>>> enough while the edits were first made, shouldn't it be enough to process
>>> them again?
>>>
>>> --Tom
>>>
>>>
>>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>> Hi Tom,
>>>>
>>>> The edits are processed sequentially, and aren't all held in memory.
>>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>>> it could resume only with remaining work if interrupted. Normally this
>>>> is not a problem in deployments given that SNN or SBN runs for
>>>> checkpointing the images and keeping the edits collection small
>>>> periodically.
>>>>
>>>> If your NameNode is running out of memory _applying_ the edits, then
>>>> the cause is not the edits but a growing namespace. You most-likely
>>>> have more files now than before, and thats going to take up permanent
>>>> memory from the NameNode heap size.
>>>>
>>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>>> > total
>>>> > (though could be expanded somewhat-- it's a VM).
>>>> >
>>>> > Though if each edit is processed sequentially (in a streaming form),
>>>> > the
>>>> > entire edits file will never be in RAM at once.
>>>> >
>>>> > Is the edits file format well defined (could I break off 100MB chunks
>>>> > and
>>>> > process them individually to achieve the same result as processing the
>>>> > whole
>>>> > thing at once)?
>>>> >
>>>> > --Tom
>>>> >
>>>> >
>>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>>> > wrote:
>>>> >>
>>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>>> >> 96Gb) should be the easiest way.
>>>> >>
>>>> >>
>>>> >> ________________________________
>>>> >> From: Tom Brown <to...@gmail.com>
>>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>>> >> Subject: Is there any way to partially process HDFS edits?
>>>> >>
>>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>>> >> larger than it should be (the secondary namenode wasn't running for
>>>> >> some
>>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>>> >>
>>>> >> The first time I tried to start the namenode, it chewed on the edits
>>>> >> for
>>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>>> >> process
>>>> >> again.
>>>> >>
>>>> >> Is there any way that the edits file can be partially processed to
>>>> >> avoid
>>>> >> having to re-process the same edits over and over until I can
>>>> >> allocate
>>>> >> enough memory for it to be done in one shot?
>>>> >>
>>>> >> How long should it take (hours? days?) to process an edits file of
>>>> >> that
>>>> >> size?
>>>> >>
>>>> >> Any help is appreciated!
>>>> >>
>>>> >> --Tom
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Tom,

That is valuable info. When we "replay" edits, we would be creating
and then deleting those files - so memory would grow in between until
the delete events begin appearing in the edit log segment.

On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> A simple estimate puts the total number of blocks somewhere around 500,000.
> Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
> that were created and quickly deleted (about 10/sec for 6 weeks) in the
> cluster, and that activity is what is contained in the edits.
>
> Since those files don't exist (quickly created and deleted), shouldn't they
> be inconsequential to the memory requirements of the namenode as it starts
> up.
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> wrote:
>>
>> Can you share how many blocks does your cluster have? how many
>> directories? how many files?
>>
>> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
>> explains how much RAM will be used for your namenode.
>> Its pretty old by hadoop version but its a good starting point.
>>
>> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
>> NameNode memory for every 1 million blocks stored in the distributed file
>> system"
>>
>> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>>
>>
>>
>> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>>>
>>> It ran again for about 15 hours before dying again. I'm seeing what extra
>>> RAM resources we can throw at this VM (maybe up to 32GB), but until then I'm
>>> trying to figure out if I'm hitting some strange bug.
>>>
>>> When the edits were originally made (over the course of 6 weeks), the
>>> namenode only had 512MB and was able to contain the filesystem completely in
>>> memory. I don't understand why it's running out of memory. If 512MB was
>>> enough while the edits were first made, shouldn't it be enough to process
>>> them again?
>>>
>>> --Tom
>>>
>>>
>>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>> Hi Tom,
>>>>
>>>> The edits are processed sequentially, and aren't all held in memory.
>>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>>> it could resume only with remaining work if interrupted. Normally this
>>>> is not a problem in deployments given that SNN or SBN runs for
>>>> checkpointing the images and keeping the edits collection small
>>>> periodically.
>>>>
>>>> If your NameNode is running out of memory _applying_ the edits, then
>>>> the cause is not the edits but a growing namespace. You most-likely
>>>> have more files now than before, and thats going to take up permanent
>>>> memory from the NameNode heap size.
>>>>
>>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>>> > total
>>>> > (though could be expanded somewhat-- it's a VM).
>>>> >
>>>> > Though if each edit is processed sequentially (in a streaming form),
>>>> > the
>>>> > entire edits file will never be in RAM at once.
>>>> >
>>>> > Is the edits file format well defined (could I break off 100MB chunks
>>>> > and
>>>> > process them individually to achieve the same result as processing the
>>>> > whole
>>>> > thing at once)?
>>>> >
>>>> > --Tom
>>>> >
>>>> >
>>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>>> > wrote:
>>>> >>
>>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>>> >> 96Gb) should be the easiest way.
>>>> >>
>>>> >>
>>>> >> ________________________________
>>>> >> From: Tom Brown <to...@gmail.com>
>>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>>> >> Subject: Is there any way to partially process HDFS edits?
>>>> >>
>>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>>> >> larger than it should be (the secondary namenode wasn't running for
>>>> >> some
>>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>>> >>
>>>> >> The first time I tried to start the namenode, it chewed on the edits
>>>> >> for
>>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>>> >> process
>>>> >> again.
>>>> >>
>>>> >> Is there any way that the edits file can be partially processed to
>>>> >> avoid
>>>> >> having to re-process the same edits over and over until I can
>>>> >> allocate
>>>> >> enough memory for it to be done in one shot?
>>>> >>
>>>> >> How long should it take (hours? days?) to process an edits file of
>>>> >> that
>>>> >> size?
>>>> >>
>>>> >> Any help is appreciated!
>>>> >>
>>>> >> --Tom
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Tom,

That is valuable info. When we "replay" edits, we would be creating
and then deleting those files - so memory would grow in between until
the delete events begin appearing in the edit log segment.

On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <to...@gmail.com> wrote:
> A simple estimate puts the total number of blocks somewhere around 500,000.
> Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
> that were created and quickly deleted (about 10/sec for 6 weeks) in the
> cluster, and that activity is what is contained in the edits.
>
> Since those files don't exist (quickly created and deleted), shouldn't they
> be inconsequential to the memory requirements of the namenode as it starts
> up.
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>
> wrote:
>>
>> Can you share how many blocks does your cluster have? how many
>> directories? how many files?
>>
>> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
>> explains how much RAM will be used for your namenode.
>> Its pretty old by hadoop version but its a good starting point.
>>
>> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
>> NameNode memory for every 1 million blocks stored in the distributed file
>> system"
>>
>> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>>
>>
>>
>> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>>>
>>> It ran again for about 15 hours before dying again. I'm seeing what extra
>>> RAM resources we can throw at this VM (maybe up to 32GB), but until then I'm
>>> trying to figure out if I'm hitting some strange bug.
>>>
>>> When the edits were originally made (over the course of 6 weeks), the
>>> namenode only had 512MB and was able to contain the filesystem completely in
>>> memory. I don't understand why it's running out of memory. If 512MB was
>>> enough while the edits were first made, shouldn't it be enough to process
>>> them again?
>>>
>>> --Tom
>>>
>>>
>>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>>>
>>>> Hi Tom,
>>>>
>>>> The edits are processed sequentially, and aren't all held in memory.
>>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>>> it could resume only with remaining work if interrupted. Normally this
>>>> is not a problem in deployments given that SNN or SBN runs for
>>>> checkpointing the images and keeping the edits collection small
>>>> periodically.
>>>>
>>>> If your NameNode is running out of memory _applying_ the edits, then
>>>> the cause is not the edits but a growing namespace. You most-likely
>>>> have more files now than before, and thats going to take up permanent
>>>> memory from the NameNode heap size.
>>>>
>>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>>> > total
>>>> > (though could be expanded somewhat-- it's a VM).
>>>> >
>>>> > Though if each edit is processed sequentially (in a streaming form),
>>>> > the
>>>> > entire edits file will never be in RAM at once.
>>>> >
>>>> > Is the edits file format well defined (could I break off 100MB chunks
>>>> > and
>>>> > process them individually to achieve the same result as processing the
>>>> > whole
>>>> > thing at once)?
>>>> >
>>>> > --Tom
>>>> >
>>>> >
>>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>>> > wrote:
>>>> >>
>>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>>> >> 96Gb) should be the easiest way.
>>>> >>
>>>> >>
>>>> >> ________________________________
>>>> >> From: Tom Brown <to...@gmail.com>
>>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>>> >> Subject: Is there any way to partially process HDFS edits?
>>>> >>
>>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>>> >> larger than it should be (the secondary namenode wasn't running for
>>>> >> some
>>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>>> >>
>>>> >> The first time I tried to start the namenode, it chewed on the edits
>>>> >> for
>>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>>> >> process
>>>> >> again.
>>>> >>
>>>> >> Is there any way that the edits file can be partially processed to
>>>> >> avoid
>>>> >> having to re-process the same edits over and over until I can
>>>> >> allocate
>>>> >> enough memory for it to be done in one shot?
>>>> >>
>>>> >> How long should it take (hours? days?) to process an edits file of
>>>> >> that
>>>> >> size?
>>>> >>
>>>> >> Any help is appreciated!
>>>> >>
>>>> >> --Tom
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

A simple estimate puts the total number of blocks somewhere around 500,000.
Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
that were created and quickly deleted (about 10/sec for 6 weeks) in the
cluster, and that activity is what is contained in the edits.

Since those files don't exist (quickly created and deleted), shouldn't they
be inconsequential to the memory requirements of the namenode as it starts
up.

--Tom


On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:

> Can you share how many blocks does your cluster have? how many
> directories? how many files?
>
> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> explains how much RAM will be used for your namenode.
> Its pretty old by hadoop version but its a good starting point.
>
> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> NameNode memory for every 1 million blocks stored in the distributed file
> system"
>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>
>
>
> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>
>> It ran again for about 15 hours before dying again. I'm seeing what extra
>> RAM resources we can throw at this VM (maybe up to 32GB), but until then
>> I'm trying to figure out if I'm hitting some strange bug.
>>
>> When the edits were originally made (over the course of 6 weeks), the
>> namenode only had 512MB and was able to contain the filesystem completely
>> in memory. I don't understand why it's running out of memory. If 512MB was
>> enough while the edits were first made, shouldn't it be enough to process
>> them again?
>>
>> --Tom
>>
>>
>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi Tom,
>>>
>>> The edits are processed sequentially, and aren't all held in memory.
>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>> it could resume only with remaining work if interrupted. Normally this
>>> is not a problem in deployments given that SNN or SBN runs for
>>> checkpointing the images and keeping the edits collection small
>>> periodically.
>>>
>>> If your NameNode is running out of memory _applying_ the edits, then
>>> the cause is not the edits but a growing namespace. You most-likely
>>> have more files now than before, and thats going to take up permanent
>>> memory from the NameNode heap size.
>>>
>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>> total
>>> > (though could be expanded somewhat-- it's a VM).
>>> >
>>> > Though if each edit is processed sequentially (in a streaming form),
>>> the
>>> > entire edits file will never be in RAM at once.
>>> >
>>> > Is the edits file format well defined (could I break off 100MB chunks
>>> and
>>> > process them individually to achieve the same result as processing the
>>> whole
>>> > thing at once)?
>>> >
>>> > --Tom
>>> >
>>> >
>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>> wrote:
>>> >>
>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>> >> 96Gb) should be the easiest way.
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Tom Brown <to...@gmail.com>
>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>> >> Subject: Is there any way to partially process HDFS edits?
>>> >>
>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>> >> larger than it should be (the secondary namenode wasn't running for
>>> some
>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>> >>
>>> >> The first time I tried to start the namenode, it chewed on the edits
>>> for
>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>> process
>>> >> again.
>>> >>
>>> >> Is there any way that the edits file can be partially processed to
>>> avoid
>>> >> having to re-process the same edits over and over until I can allocate
>>> >> enough memory for it to be done in one shot?
>>> >>
>>> >> How long should it take (hours? days?) to process an edits file of
>>> that
>>> >> size?
>>> >>
>>> >> Any help is appreciated!
>>> >>
>>> >> --Tom
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

A simple estimate puts the total number of blocks somewhere around 500,000.
Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
that were created and quickly deleted (about 10/sec for 6 weeks) in the
cluster, and that activity is what is contained in the edits.

Since those files don't exist (quickly created and deleted), shouldn't they
be inconsequential to the memory requirements of the namenode as it starts
up.

--Tom


On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:

> Can you share how many blocks does your cluster have? how many
> directories? how many files?
>
> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> explains how much RAM will be used for your namenode.
> Its pretty old by hadoop version but its a good starting point.
>
> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> NameNode memory for every 1 million blocks stored in the distributed file
> system"
>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>
>
>
> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>
>> It ran again for about 15 hours before dying again. I'm seeing what extra
>> RAM resources we can throw at this VM (maybe up to 32GB), but until then
>> I'm trying to figure out if I'm hitting some strange bug.
>>
>> When the edits were originally made (over the course of 6 weeks), the
>> namenode only had 512MB and was able to contain the filesystem completely
>> in memory. I don't understand why it's running out of memory. If 512MB was
>> enough while the edits were first made, shouldn't it be enough to process
>> them again?
>>
>> --Tom
>>
>>
>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi Tom,
>>>
>>> The edits are processed sequentially, and aren't all held in memory.
>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>> it could resume only with remaining work if interrupted. Normally this
>>> is not a problem in deployments given that SNN or SBN runs for
>>> checkpointing the images and keeping the edits collection small
>>> periodically.
>>>
>>> If your NameNode is running out of memory _applying_ the edits, then
>>> the cause is not the edits but a growing namespace. You most-likely
>>> have more files now than before, and thats going to take up permanent
>>> memory from the NameNode heap size.
>>>
>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>> total
>>> > (though could be expanded somewhat-- it's a VM).
>>> >
>>> > Though if each edit is processed sequentially (in a streaming form),
>>> the
>>> > entire edits file will never be in RAM at once.
>>> >
>>> > Is the edits file format well defined (could I break off 100MB chunks
>>> and
>>> > process them individually to achieve the same result as processing the
>>> whole
>>> > thing at once)?
>>> >
>>> > --Tom
>>> >
>>> >
>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>> wrote:
>>> >>
>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>> >> 96Gb) should be the easiest way.
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Tom Brown <to...@gmail.com>
>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>> >> Subject: Is there any way to partially process HDFS edits?
>>> >>
>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>> >> larger than it should be (the secondary namenode wasn't running for
>>> some
>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>> >>
>>> >> The first time I tried to start the namenode, it chewed on the edits
>>> for
>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>> process
>>> >> again.
>>> >>
>>> >> Is there any way that the edits file can be partially processed to
>>> avoid
>>> >> having to re-process the same edits over and over until I can allocate
>>> >> enough memory for it to be done in one shot?
>>> >>
>>> >> How long should it take (hours? days?) to process an edits file of
>>> that
>>> >> size?
>>> >>
>>> >> Any help is appreciated!
>>> >>
>>> >> --Tom
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

A simple estimate puts the total number of blocks somewhere around 500,000.
Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
that were created and quickly deleted (about 10/sec for 6 weeks) in the
cluster, and that activity is what is contained in the edits.

Since those files don't exist (quickly created and deleted), shouldn't they
be inconsequential to the memory requirements of the namenode as it starts
up.

--Tom


On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:

> Can you share how many blocks does your cluster have? how many
> directories? how many files?
>
> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> explains how much RAM will be used for your namenode.
> Its pretty old by hadoop version but its a good starting point.
>
> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> NameNode memory for every 1 million blocks stored in the distributed file
> system"
>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>
>
>
> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>
>> It ran again for about 15 hours before dying again. I'm seeing what extra
>> RAM resources we can throw at this VM (maybe up to 32GB), but until then
>> I'm trying to figure out if I'm hitting some strange bug.
>>
>> When the edits were originally made (over the course of 6 weeks), the
>> namenode only had 512MB and was able to contain the filesystem completely
>> in memory. I don't understand why it's running out of memory. If 512MB was
>> enough while the edits were first made, shouldn't it be enough to process
>> them again?
>>
>> --Tom
>>
>>
>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi Tom,
>>>
>>> The edits are processed sequentially, and aren't all held in memory.
>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>> it could resume only with remaining work if interrupted. Normally this
>>> is not a problem in deployments given that SNN or SBN runs for
>>> checkpointing the images and keeping the edits collection small
>>> periodically.
>>>
>>> If your NameNode is running out of memory _applying_ the edits, then
>>> the cause is not the edits but a growing namespace. You most-likely
>>> have more files now than before, and thats going to take up permanent
>>> memory from the NameNode heap size.
>>>
>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>> total
>>> > (though could be expanded somewhat-- it's a VM).
>>> >
>>> > Though if each edit is processed sequentially (in a streaming form),
>>> the
>>> > entire edits file will never be in RAM at once.
>>> >
>>> > Is the edits file format well defined (could I break off 100MB chunks
>>> and
>>> > process them individually to achieve the same result as processing the
>>> whole
>>> > thing at once)?
>>> >
>>> > --Tom
>>> >
>>> >
>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>> wrote:
>>> >>
>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>> >> 96Gb) should be the easiest way.
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Tom Brown <to...@gmail.com>
>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>> >> Subject: Is there any way to partially process HDFS edits?
>>> >>
>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>> >> larger than it should be (the secondary namenode wasn't running for
>>> some
>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>> >>
>>> >> The first time I tried to start the namenode, it chewed on the edits
>>> for
>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>> process
>>> >> again.
>>> >>
>>> >> Is there any way that the edits file can be partially processed to
>>> avoid
>>> >> having to re-process the same edits over and over until I can allocate
>>> >> enough memory for it to be done in one shot?
>>> >>
>>> >> How long should it take (hours? days?) to process an edits file of
>>> that
>>> >> size?
>>> >>
>>> >> Any help is appreciated!
>>> >>
>>> >> --Tom
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

A simple estimate puts the total number of blocks somewhere around 500,000.
Due to an HBase bug (HBASE-9648), there were approximately 50,000,000 files
that were created and quickly deleted (about 10/sec for 6 weeks) in the
cluster, and that activity is what is contained in the edits.

Since those files don't exist (quickly created and deleted), shouldn't they
be inconsequential to the memory requirements of the namenode as it starts
up.

--Tom


On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:

> Can you share how many blocks does your cluster have? how many
> directories? how many files?
>
> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> explains how much RAM will be used for your namenode.
> Its pretty old by hadoop version but its a good starting point.
>
> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> NameNode memory for every 1 million blocks stored in the distributed file
> system"
>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
>
>
>
> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:
>
>> It ran again for about 15 hours before dying again. I'm seeing what extra
>> RAM resources we can throw at this VM (maybe up to 32GB), but until then
>> I'm trying to figure out if I'm hitting some strange bug.
>>
>> When the edits were originally made (over the course of 6 weeks), the
>> namenode only had 512MB and was able to contain the filesystem completely
>> in memory. I don't understand why it's running out of memory. If 512MB was
>> enough while the edits were first made, shouldn't it be enough to process
>> them again?
>>
>> --Tom
>>
>>
>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi Tom,
>>>
>>> The edits are processed sequentially, and aren't all held in memory.
>>> Right now there's no mid-way-checkpoint when it is loaded, such that
>>> it could resume only with remaining work if interrupted. Normally this
>>> is not a problem in deployments given that SNN or SBN runs for
>>> checkpointing the images and keeping the edits collection small
>>> periodically.
>>>
>>> If your NameNode is running out of memory _applying_ the edits, then
>>> the cause is not the edits but a growing namespace. You most-likely
>>> have more files now than before, and thats going to take up permanent
>>> memory from the NameNode heap size.
>>>
>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB
>>> total
>>> > (though could be expanded somewhat-- it's a VM).
>>> >
>>> > Though if each edit is processed sequentially (in a streaming form),
>>> the
>>> > entire edits file will never be in RAM at once.
>>> >
>>> > Is the edits file format well defined (could I break off 100MB chunks
>>> and
>>> > process them individually to achieve the same result as processing the
>>> whole
>>> > thing at once)?
>>> >
>>> > --Tom
>>> >
>>> >
>>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>>> wrote:
>>> >>
>>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>>> >> 96Gb) should be the easiest way.
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Tom Brown <to...@gmail.com>
>>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>>> >> Subject: Is there any way to partially process HDFS edits?
>>> >>
>>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>>> >> larger than it should be (the secondary namenode wasn't running for
>>> some
>>> >> time, and HBASE-9648 caused a huge number of additional edits).
>>> >>
>>> >> The first time I tried to start the namenode, it chewed on the edits
>>> for
>>> >> about 4 hours and then ran out of memory. I have increased the memory
>>> >> available to the namenode (was 512MB, now 2GB), and started the
>>> process
>>> >> again.
>>> >>
>>> >> Is there any way that the edits file can be partially processed to
>>> avoid
>>> >> having to re-process the same edits over and over until I can allocate
>>> >> enough memory for it to be done in one shot?
>>> >>
>>> >> How long should it take (hours? days?) to process an edits file of
>>> that
>>> >> size?
>>> >>
>>> >> Any help is appreciated!
>>> >>
>>> >> --Tom
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Is there any way to partially process HDFS edits?

[Nitin Pawar <ni...@gmail.com>]

Can you share how many blocks does your cluster have? how many directories?
how many files?

There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
explains how much RAM will be used for your namenode.
Its pretty old by hadoop version but its a good starting point.

According to Cloudera's blog "A good rule of thumb is to assume 1GB of
NameNode memory for every 1 million blocks stored in the distributed file
system"
http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/



On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:

> It ran again for about 15 hours before dying again. I'm seeing what extra
> RAM resources we can throw at this VM (maybe up to 32GB), but until then
> I'm trying to figure out if I'm hitting some strange bug.
>
> When the edits were originally made (over the course of 6 weeks), the
> namenode only had 512MB and was able to contain the filesystem completely
> in memory. I don't understand why it's running out of memory. If 512MB was
> enough while the edits were first made, shouldn't it be enough to process
> them again?
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>


-- 
Nitin Pawar

Re: Is there any way to partially process HDFS edits?

[Nitin Pawar <ni...@gmail.com>]

Can you share how many blocks does your cluster have? how many directories?
how many files?

There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
explains how much RAM will be used for your namenode.
Its pretty old by hadoop version but its a good starting point.

According to Cloudera's blog "A good rule of thumb is to assume 1GB of
NameNode memory for every 1 million blocks stored in the distributed file
system"
http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/



On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:

> It ran again for about 15 hours before dying again. I'm seeing what extra
> RAM resources we can throw at this VM (maybe up to 32GB), but until then
> I'm trying to figure out if I'm hitting some strange bug.
>
> When the edits were originally made (over the course of 6 weeks), the
> namenode only had 512MB and was able to contain the filesystem completely
> in memory. I don't understand why it's running out of memory. If 512MB was
> enough while the edits were first made, shouldn't it be enough to process
> them again?
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>


-- 
Nitin Pawar

Re: Is there any way to partially process HDFS edits?

[Nitin Pawar <ni...@gmail.com>]

Can you share how many blocks does your cluster have? how many directories?
how many files?

There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
explains how much RAM will be used for your namenode.
Its pretty old by hadoop version but its a good starting point.

According to Cloudera's blog "A good rule of thumb is to assume 1GB of
NameNode memory for every 1 million blocks stored in the distributed file
system"
http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/



On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:

> It ran again for about 15 hours before dying again. I'm seeing what extra
> RAM resources we can throw at this VM (maybe up to 32GB), but until then
> I'm trying to figure out if I'm hitting some strange bug.
>
> When the edits were originally made (over the course of 6 weeks), the
> namenode only had 512MB and was able to contain the filesystem completely
> in memory. I don't understand why it's running out of memory. If 512MB was
> enough while the edits were first made, shouldn't it be enough to process
> them again?
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>


-- 
Nitin Pawar

Re: Is there any way to partially process HDFS edits?

[Nitin Pawar <ni...@gmail.com>]

Can you share how many blocks does your cluster have? how many directories?
how many files?

There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
explains how much RAM will be used for your namenode.
Its pretty old by hadoop version but its a good starting point.

According to Cloudera's blog "A good rule of thumb is to assume 1GB of
NameNode memory for every 1 million blocks stored in the distributed file
system"
http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/



On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <to...@gmail.com> wrote:

> It ran again for about 15 hours before dying again. I'm seeing what extra
> RAM resources we can throw at this VM (maybe up to 32GB), but until then
> I'm trying to figure out if I'm hitting some strange bug.
>
> When the edits were originally made (over the course of 6 weeks), the
> namenode only had 512MB and was able to contain the filesystem completely
> in memory. I don't understand why it's running out of memory. If 512MB was
> enough while the edits were first made, shouldn't it be enough to process
> them again?
>
> --Tom
>
>
> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>


-- 
Nitin Pawar

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

It ran again for about 15 hours before dying again. I'm seeing what extra
RAM resources we can throw at this VM (maybe up to 32GB), but until then
I'm trying to figure out if I'm hitting some strange bug.

When the edits were originally made (over the course of 6 weeks), the
namenode only had 512MB and was able to contain the filesystem completely
in memory. I don't understand why it's running out of memory. If 512MB was
enough while the edits were first made, shouldn't it be enough to process
them again?

--Tom


On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

It ran again for about 15 hours before dying again. I'm seeing what extra
RAM resources we can throw at this VM (maybe up to 32GB), but until then
I'm trying to figure out if I'm hitting some strange bug.

When the edits were originally made (over the course of 6 weeks), the
namenode only had 512MB and was able to contain the filesystem completely
in memory. I don't understand why it's running out of memory. If 512MB was
enough while the edits were first made, shouldn't it be enough to process
them again?

--Tom


On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Shashwat,

When you say "manual checkpointing", are you talking about a specific
process? Or are you just recommending that I run the secondarynamenode to
prevent this situation in the first place?

--Tom


On Thu, Sep 26, 2013 at 6:29 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:

> Just try to do mannual checkpointing..
>
>
> *Thanks & Regards    *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Shashwat,

When you say "manual checkpointing", are you talking about a specific
process? Or are you just recommending that I run the secondarynamenode to
prevent this situation in the first place?

--Tom


On Thu, Sep 26, 2013 at 6:29 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:

> Just try to do mannual checkpointing..
>
>
> *Thanks & Regards    *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Shashwat,

When you say "manual checkpointing", are you talking about a specific
process? Or are you just recommending that I run the secondarynamenode to
prevent this situation in the first place?

--Tom


On Thu, Sep 26, 2013 at 6:29 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:

> Just try to do mannual checkpointing..
>
>
> *Thanks & Regards    *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Shashwat,

When you say "manual checkpointing", are you talking about a specific
process? Or are you just recommending that I run the secondarynamenode to
prevent this situation in the first place?

--Tom


On Thu, Sep 26, 2013 at 6:29 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:

> Just try to do mannual checkpointing..
>
>
> *Thanks & Regards    *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi Tom,
>>
>> The edits are processed sequentially, and aren't all held in memory.
>> Right now there's no mid-way-checkpoint when it is loaded, such that
>> it could resume only with remaining work if interrupted. Normally this
>> is not a problem in deployments given that SNN or SBN runs for
>> checkpointing the images and keeping the edits collection small
>> periodically.
>>
>> If your NameNode is running out of memory _applying_ the edits, then
>> the cause is not the edits but a growing namespace. You most-likely
>> have more files now than before, and thats going to take up permanent
>> memory from the NameNode heap size.
>>
>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
>> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
>> > (though could be expanded somewhat-- it's a VM).
>> >
>> > Though if each edit is processed sequentially (in a streaming form), the
>> > entire edits file will never be in RAM at once.
>> >
>> > Is the edits file format well defined (could I break off 100MB chunks
>> and
>> > process them individually to achieve the same result as processing the
>> whole
>> > thing at once)?
>> >
>> > --Tom
>> >
>> >
>> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com>
>> wrote:
>> >>
>> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> >> 96Gb) should be the easiest way.
>> >>
>> >>
>> >> ________________________________
>> >> From: Tom Brown <to...@gmail.com>
>> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> >> Sent: Wednesday, September 25, 2013 11:29 AM
>> >> Subject: Is there any way to partially process HDFS edits?
>> >>
>> >> I have an edits file on my namenode that is 35GB. This is quite a bit
>> >> larger than it should be (the secondary namenode wasn't running for
>> some
>> >> time, and HBASE-9648 caused a huge number of additional edits).
>> >>
>> >> The first time I tried to start the namenode, it chewed on the edits
>> for
>> >> about 4 hours and then ran out of memory. I have increased the memory
>> >> available to the namenode (was 512MB, now 2GB), and started the process
>> >> again.
>> >>
>> >> Is there any way that the edits file can be partially processed to
>> avoid
>> >> having to re-process the same edits over and over until I can allocate
>> >> enough memory for it to be done in one shot?
>> >>
>> >> How long should it take (hours? days?) to process an edits file of that
>> >> size?
>> >>
>> >> Any help is appreciated!
>> >>
>> >> --Tom
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Is there any way to partially process HDFS edits?

[shashwat shriparv <dw...@gmail.com>]

Just try to do mannual checkpointing..


*Thanks & Regards    *

∞
Shashwat Shriparv



On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[shashwat shriparv <dw...@gmail.com>]

Just try to do mannual checkpointing..


*Thanks & Regards    *

∞
Shashwat Shriparv



On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[shashwat shriparv <dw...@gmail.com>]

Just try to do mannual checkpointing..


*Thanks & Regards    *

∞
Shashwat Shriparv



On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[shashwat shriparv <dw...@gmail.com>]

Just try to do mannual checkpointing..


*Thanks & Regards    *

∞
Shashwat Shriparv



On Thu, Sep 26, 2013 at 5:35 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

It ran again for about 15 hours before dying again. I'm seeing what extra
RAM resources we can throw at this VM (maybe up to 32GB), but until then
I'm trying to figure out if I'm hitting some strange bug.

When the edits were originally made (over the course of 6 weeks), the
namenode only had 512MB and was able to contain the filesystem completely
in memory. I don't understand why it's running out of memory. If 512MB was
enough while the edits were first made, shouldn't it be enough to process
them again?

--Tom


On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

It ran again for about 15 hours before dying again. I'm seeing what extra
RAM resources we can throw at this VM (maybe up to 32GB), but until then
I'm trying to figure out if I'm hitting some strange bug.

When the edits were originally made (over the course of 6 weeks), the
namenode only had 512MB and was able to contain the filesystem completely
in memory. I don't understand why it's running out of memory. If 512MB was
enough while the edits were first made, shouldn't it be enough to process
them again?

--Tom


On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <ha...@cloudera.com> wrote:

> Hi Tom,
>
> The edits are processed sequentially, and aren't all held in memory.
> Right now there's no mid-way-checkpoint when it is loaded, such that
> it could resume only with remaining work if interrupted. Normally this
> is not a problem in deployments given that SNN or SBN runs for
> checkpointing the images and keeping the edits collection small
> periodically.
>
> If your NameNode is running out of memory _applying_ the edits, then
> the cause is not the edits but a growing namespace. You most-likely
> have more files now than before, and thats going to take up permanent
> memory from the NameNode heap size.
>
> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> > Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> > (though could be expanded somewhat-- it's a VM).
> >
> > Though if each edit is processed sequentially (in a streaming form), the
> > entire edits file will never be in RAM at once.
> >
> > Is the edits file format well defined (could I break off 100MB chunks and
> > process them individually to achieve the same result as processing the
> whole
> > thing at once)?
> >
> > --Tom
> >
> >
> > On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
> >>
> >> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> >> 96Gb) should be the easiest way.
> >>
> >>
> >> ________________________________
> >> From: Tom Brown <to...@gmail.com>
> >> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> >> Sent: Wednesday, September 25, 2013 11:29 AM
> >> Subject: Is there any way to partially process HDFS edits?
> >>
> >> I have an edits file on my namenode that is 35GB. This is quite a bit
> >> larger than it should be (the secondary namenode wasn't running for some
> >> time, and HBASE-9648 caused a huge number of additional edits).
> >>
> >> The first time I tried to start the namenode, it chewed on the edits for
> >> about 4 hours and then ran out of memory. I have increased the memory
> >> available to the namenode (was 512MB, now 2GB), and started the process
> >> again.
> >>
> >> Is there any way that the edits file can be partially processed to avoid
> >> having to re-process the same edits over and over until I can allocate
> >> enough memory for it to be done in one shot?
> >>
> >> How long should it take (hours? days?) to process an edits file of that
> >> size?
> >>
> >> Any help is appreciated!
> >>
> >> --Tom
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Hi Tom,

The edits are processed sequentially, and aren't all held in memory.
Right now there's no mid-way-checkpoint when it is loaded, such that
it could resume only with remaining work if interrupted. Normally this
is not a problem in deployments given that SNN or SBN runs for
checkpointing the images and keeping the edits collection small
periodically.

If your NameNode is running out of memory _applying_ the edits, then
the cause is not the edits but a growing namespace. You most-likely
have more files now than before, and thats going to take up permanent
memory from the NameNode heap size.

On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> (though could be expanded somewhat-- it's a VM).
>
> Though if each edit is processed sequentially (in a streaming form), the
> entire edits file will never be in RAM at once.
>
> Is the edits file format well defined (could I break off 100MB chunks and
> process them individually to achieve the same result as processing the whole
> thing at once)?
>
> --Tom
>
>
> On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
>>
>> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> 96Gb) should be the easiest way.
>>
>>
>> ________________________________
>> From: Tom Brown <to...@gmail.com>
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Sent: Wednesday, September 25, 2013 11:29 AM
>> Subject: Is there any way to partially process HDFS edits?
>>
>> I have an edits file on my namenode that is 35GB. This is quite a bit
>> larger than it should be (the secondary namenode wasn't running for some
>> time, and HBASE-9648 caused a huge number of additional edits).
>>
>> The first time I tried to start the namenode, it chewed on the edits for
>> about 4 hours and then ran out of memory. I have increased the memory
>> available to the namenode (was 512MB, now 2GB), and started the process
>> again.
>>
>> Is there any way that the edits file can be partially processed to avoid
>> having to re-process the same edits over and over until I can allocate
>> enough memory for it to be done in one shot?
>>
>> How long should it take (hours? days?) to process an edits file of that
>> size?
>>
>> Any help is appreciated!
>>
>> --Tom
>>
>>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Hi Tom,

The edits are processed sequentially, and aren't all held in memory.
Right now there's no mid-way-checkpoint when it is loaded, such that
it could resume only with remaining work if interrupted. Normally this
is not a problem in deployments given that SNN or SBN runs for
checkpointing the images and keeping the edits collection small
periodically.

If your NameNode is running out of memory _applying_ the edits, then
the cause is not the edits but a growing namespace. You most-likely
have more files now than before, and thats going to take up permanent
memory from the NameNode heap size.

On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> (though could be expanded somewhat-- it's a VM).
>
> Though if each edit is processed sequentially (in a streaming form), the
> entire edits file will never be in RAM at once.
>
> Is the edits file format well defined (could I break off 100MB chunks and
> process them individually to achieve the same result as processing the whole
> thing at once)?
>
> --Tom
>
>
> On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
>>
>> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> 96Gb) should be the easiest way.
>>
>>
>> ________________________________
>> From: Tom Brown <to...@gmail.com>
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Sent: Wednesday, September 25, 2013 11:29 AM
>> Subject: Is there any way to partially process HDFS edits?
>>
>> I have an edits file on my namenode that is 35GB. This is quite a bit
>> larger than it should be (the secondary namenode wasn't running for some
>> time, and HBASE-9648 caused a huge number of additional edits).
>>
>> The first time I tried to start the namenode, it chewed on the edits for
>> about 4 hours and then ran out of memory. I have increased the memory
>> available to the namenode (was 512MB, now 2GB), and started the process
>> again.
>>
>> Is there any way that the edits file can be partially processed to avoid
>> having to re-process the same edits over and over until I can allocate
>> enough memory for it to be done in one shot?
>>
>> How long should it take (hours? days?) to process an edits file of that
>> size?
>>
>> Any help is appreciated!
>>
>> --Tom
>>
>>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Hi Tom,

The edits are processed sequentially, and aren't all held in memory.
Right now there's no mid-way-checkpoint when it is loaded, such that
it could resume only with remaining work if interrupted. Normally this
is not a problem in deployments given that SNN or SBN runs for
checkpointing the images and keeping the edits collection small
periodically.

If your NameNode is running out of memory _applying_ the edits, then
the cause is not the edits but a growing namespace. You most-likely
have more files now than before, and thats going to take up permanent
memory from the NameNode heap size.

On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> (though could be expanded somewhat-- it's a VM).
>
> Though if each edit is processed sequentially (in a streaming form), the
> entire edits file will never be in RAM at once.
>
> Is the edits file format well defined (could I break off 100MB chunks and
> process them individually to achieve the same result as processing the whole
> thing at once)?
>
> --Tom
>
>
> On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
>>
>> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> 96Gb) should be the easiest way.
>>
>>
>> ________________________________
>> From: Tom Brown <to...@gmail.com>
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Sent: Wednesday, September 25, 2013 11:29 AM
>> Subject: Is there any way to partially process HDFS edits?
>>
>> I have an edits file on my namenode that is 35GB. This is quite a bit
>> larger than it should be (the secondary namenode wasn't running for some
>> time, and HBASE-9648 caused a huge number of additional edits).
>>
>> The first time I tried to start the namenode, it chewed on the edits for
>> about 4 hours and then ran out of memory. I have increased the memory
>> available to the namenode (was 512MB, now 2GB), and started the process
>> again.
>>
>> Is there any way that the edits file can be partially processed to avoid
>> having to re-process the same edits over and over until I can allocate
>> enough memory for it to be done in one shot?
>>
>> How long should it take (hours? days?) to process an edits file of that
>> size?
>>
>> Any help is appreciated!
>>
>> --Tom
>>
>>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Harsh J <ha...@cloudera.com>]

Hi Tom,

The edits are processed sequentially, and aren't all held in memory.
Right now there's no mid-way-checkpoint when it is loaded, such that
it could resume only with remaining work if interrupted. Normally this
is not a problem in deployments given that SNN or SBN runs for
checkpointing the images and keeping the edits collection small
periodically.

If your NameNode is running out of memory _applying_ the edits, then
the cause is not the edits but a growing namespace. You most-likely
have more files now than before, and thats going to take up permanent
memory from the NameNode heap size.

On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <to...@gmail.com> wrote:
> Unfortunately, I cannot give it that much RAM. The machine has 4GB total
> (though could be expanded somewhat-- it's a VM).
>
> Though if each edit is processed sequentially (in a streaming form), the
> entire edits file will never be in RAM at once.
>
> Is the edits file format well defined (could I break off 100MB chunks and
> process them individually to achieve the same result as processing the whole
> thing at once)?
>
> --Tom
>
>
> On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:
>>
>> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
>> 96Gb) should be the easiest way.
>>
>>
>> ________________________________
>> From: Tom Brown <to...@gmail.com>
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Sent: Wednesday, September 25, 2013 11:29 AM
>> Subject: Is there any way to partially process HDFS edits?
>>
>> I have an edits file on my namenode that is 35GB. This is quite a bit
>> larger than it should be (the secondary namenode wasn't running for some
>> time, and HBASE-9648 caused a huge number of additional edits).
>>
>> The first time I tried to start the namenode, it chewed on the edits for
>> about 4 hours and then ran out of memory. I have increased the memory
>> available to the namenode (was 512MB, now 2GB), and started the process
>> again.
>>
>> Is there any way that the edits file can be partially processed to avoid
>> having to re-process the same edits over and over until I can allocate
>> enough memory for it to be done in one shot?
>>
>> How long should it take (hours? days?) to process an edits file of that
>> size?
>>
>> Any help is appreciated!
>>
>> --Tom
>>
>>
>



-- 
Harsh J

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Unfortunately, I cannot give it that much RAM. The machine has 4GB total
(though could be expanded somewhat-- it's a VM).

Though if each edit is processed sequentially (in a streaming form), the
entire edits file will never be in RAM at once.

Is the edits file format well defined (could I break off 100MB chunks and
process them individually to achieve the same result as processing the
whole thing at once)?

--Tom


On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:

> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> 96Gb) should be the easiest way.
>
>
>   ------------------------------
>  *From:* Tom Brown <to...@gmail.com>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Wednesday, September 25, 2013 11:29 AM
> *Subject:* Is there any way to partially process HDFS edits?
>
> I have an edits file on my namenode that is 35GB. This is quite a bit
> larger than it should be (the secondary namenode wasn't running for some
> time, and HBASE-9648 caused a huge number of additional edits).
>
> The first time I tried to start the namenode, it chewed on the edits for
> about 4 hours and then ran out of memory. I have increased the memory
> available to the namenode (was 512MB, now 2GB), and started the process
> again.
>
> Is there any way that the edits file can be partially processed to avoid
> having to re-process the same edits over and over until I can allocate
> enough memory for it to be done in one shot?
>
> How long should it take (hours? days?) to process an edits file of that
> size?
>
> Any help is appreciated!
>
> --Tom
>
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Unfortunately, I cannot give it that much RAM. The machine has 4GB total
(though could be expanded somewhat-- it's a VM).

Though if each edit is processed sequentially (in a streaming form), the
entire edits file will never be in RAM at once.

Is the edits file format well defined (could I break off 100MB chunks and
process them individually to achieve the same result as processing the
whole thing at once)?

--Tom


On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:

> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> 96Gb) should be the easiest way.
>
>
>   ------------------------------
>  *From:* Tom Brown <to...@gmail.com>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Wednesday, September 25, 2013 11:29 AM
> *Subject:* Is there any way to partially process HDFS edits?
>
> I have an edits file on my namenode that is 35GB. This is quite a bit
> larger than it should be (the secondary namenode wasn't running for some
> time, and HBASE-9648 caused a huge number of additional edits).
>
> The first time I tried to start the namenode, it chewed on the edits for
> about 4 hours and then ran out of memory. I have increased the memory
> available to the namenode (was 512MB, now 2GB), and started the process
> again.
>
> Is there any way that the edits file can be partially processed to avoid
> having to re-process the same edits over and over until I can allocate
> enough memory for it to be done in one shot?
>
> How long should it take (hours? days?) to process an edits file of that
> size?
>
> Any help is appreciated!
>
> --Tom
>
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Unfortunately, I cannot give it that much RAM. The machine has 4GB total
(though could be expanded somewhat-- it's a VM).

Though if each edit is processed sequentially (in a streaming form), the
entire edits file will never be in RAM at once.

Is the edits file format well defined (could I break off 100MB chunks and
process them individually to achieve the same result as processing the
whole thing at once)?

--Tom


On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:

> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> 96Gb) should be the easiest way.
>
>
>   ------------------------------
>  *From:* Tom Brown <to...@gmail.com>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Wednesday, September 25, 2013 11:29 AM
> *Subject:* Is there any way to partially process HDFS edits?
>
> I have an edits file on my namenode that is 35GB. This is quite a bit
> larger than it should be (the secondary namenode wasn't running for some
> time, and HBASE-9648 caused a huge number of additional edits).
>
> The first time I tried to start the namenode, it chewed on the edits for
> about 4 hours and then ran out of memory. I have increased the memory
> available to the namenode (was 512MB, now 2GB), and started the process
> again.
>
> Is there any way that the edits file can be partially processed to avoid
> having to re-process the same edits over and over until I can allocate
> enough memory for it to be done in one shot?
>
> How long should it take (hours? days?) to process an edits file of that
> size?
>
> Any help is appreciated!
>
> --Tom
>
>
>

Re: Is there any way to partially process HDFS edits?

[Tom Brown <to...@gmail.com>]

Unfortunately, I cannot give it that much RAM. The machine has 4GB total
(though could be expanded somewhat-- it's a VM).

Though if each edit is processed sequentially (in a streaming form), the
entire edits file will never be in RAM at once.

Is the edits file format well defined (could I break off 100MB chunks and
process them individually to achieve the same result as processing the
whole thing at once)?

--Tom


On Wed, Sep 25, 2013 at 1:53 PM, Ravi Prakash <ra...@ymail.com> wrote:

> Tom! I would guess that just giving the NN JVM lots of memory (64Gb /
> 96Gb) should be the easiest way.
>
>
>   ------------------------------
>  *From:* Tom Brown <to...@gmail.com>
> *To:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Sent:* Wednesday, September 25, 2013 11:29 AM
> *Subject:* Is there any way to partially process HDFS edits?
>
> I have an edits file on my namenode that is 35GB. This is quite a bit
> larger than it should be (the secondary namenode wasn't running for some
> time, and HBASE-9648 caused a huge number of additional edits).
>
> The first time I tried to start the namenode, it chewed on the edits for
> about 4 hours and then ran out of memory. I have increased the memory
> available to the namenode (was 512MB, now 2GB), and started the process
> again.
>
> Is there any way that the edits file can be partially processed to avoid
> having to re-process the same edits over and over until I can allocate
> enough memory for it to be done in one shot?
>
> How long should it take (hours? days?) to process an edits file of that
> size?
>
> Any help is appreciated!
>
> --Tom
>
>
>

Re: Is there any way to partially process HDFS edits?

[Ravi Prakash <ra...@ymail.com>]

Tom! I would guess that just giving the NN JVM lots of memory (64Gb / 96Gb) should be the easiest way.




________________________________
 From: Tom Brown <to...@gmail.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Wednesday, September 25, 2013 11:29 AM
Subject: Is there any way to partially process HDFS edits?
 


I have an edits file on my namenode that is 35GB. This is quite a bit larger than it should be (the secondary namenode wasn't running for some time, and HBASE-9648 caused a huge number of additional edits).

The first time I tried to start the namenode, it chewed on the edits for about 4 hours and then ran out of memory. I have increased the memory available to the namenode (was 512MB, now 2GB), and started the process again.

Is there any way that the edits file can be partially processed to avoid having to re-process the same edits over and over until I can allocate enough memory for it to be done in one shot?


How long should it take (hours? days?) to process an edits file of that size?

Any help is appreciated!

--Tom

Re: Is there any way to partially process HDFS edits?

[Ravi Prakash <ra...@ymail.com>]

Tom! I would guess that just giving the NN JVM lots of memory (64Gb / 96Gb) should be the easiest way.




________________________________
 From: Tom Brown <to...@gmail.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Wednesday, September 25, 2013 11:29 AM
Subject: Is there any way to partially process HDFS edits?
 


I have an edits file on my namenode that is 35GB. This is quite a bit larger than it should be (the secondary namenode wasn't running for some time, and HBASE-9648 caused a huge number of additional edits).

The first time I tried to start the namenode, it chewed on the edits for about 4 hours and then ran out of memory. I have increased the memory available to the namenode (was 512MB, now 2GB), and started the process again.

Is there any way that the edits file can be partially processed to avoid having to re-process the same edits over and over until I can allocate enough memory for it to be done in one shot?


How long should it take (hours? days?) to process an edits file of that size?

Any help is appreciated!

--Tom

Re: Is there any way to partially process HDFS edits?

[Ravi Prakash <ra...@ymail.com>]

Tom! I would guess that just giving the NN JVM lots of memory (64Gb / 96Gb) should be the easiest way.




________________________________
 From: Tom Brown <to...@gmail.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Wednesday, September 25, 2013 11:29 AM
Subject: Is there any way to partially process HDFS edits?
 


I have an edits file on my namenode that is 35GB. This is quite a bit larger than it should be (the secondary namenode wasn't running for some time, and HBASE-9648 caused a huge number of additional edits).

The first time I tried to start the namenode, it chewed on the edits for about 4 hours and then ran out of memory. I have increased the memory available to the namenode (was 512MB, now 2GB), and started the process again.

Is there any way that the edits file can be partially processed to avoid having to re-process the same edits over and over until I can allocate enough memory for it to be done in one shot?


How long should it take (hours? days?) to process an edits file of that size?

Any help is appreciated!

--Tom

Re: Is there any way to partially process HDFS edits?

[Ravi Prakash <ra...@ymail.com>]

Tom! I would guess that just giving the NN JVM lots of memory (64Gb / 96Gb) should be the easiest way.




________________________________
 From: Tom Brown <to...@gmail.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Sent: Wednesday, September 25, 2013 11:29 AM
Subject: Is there any way to partially process HDFS edits?
 


I have an edits file on my namenode that is 35GB. This is quite a bit larger than it should be (the secondary namenode wasn't running for some time, and HBASE-9648 caused a huge number of additional edits).

The first time I tried to start the namenode, it chewed on the edits for about 4 hours and then ran out of memory. I have increased the memory available to the namenode (was 512MB, now 2GB), and started the process again.

Is there any way that the edits file can be partially processed to avoid having to re-process the same edits over and over until I can allocate enough memory for it to be done in one shot?


How long should it take (hours? days?) to process an edits file of that size?

Any help is appreciated!

--Tom