You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jim Kellerman <ji...@powerset.com> on 2008/01/04 00:20:41 UTC
Question for HBase users
Do you have data stored in HBase that you cannot recreate?
HADOOP-2478 will introduce an incompatible change in how HBase
lays out files in HDFS so that should the root or meta tables
be corrupted, it will be possible to reconstruct them from
information in the file system alone.
The problem is in building a migration utility. Anything that
we could build to migrate from the current file structure to
the new file structure would require that the root an meta
regions be absolutely correct. If they are not, the migration
would fail, because there is not enough information on disk
currently to rebuild the root and meta regions.
Is it acceptable for this change to be made without the provision
of an upgrade utility?
If not, are you willing to accept the risk that the upgrade
may fail if you have corruption in your root or meta regions?
After HADOOP-2478, we will be able to build a fault tolerant
upgrade utility, should HBase's file structure change again.
Additionally, we will be able to provide the equivalent of
fsck for HBase after HADOOP-2478.
---
Jim Kellerman, Senior Engineer; Powerset
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date: 1/2/2008 11:29 AM
Re: Question for HBase users
Posted by Billy <sa...@pearsonwholesale.com>.
Saying that the working release only came out 3 months ago I would hope no
one had data stored in hbase at this time that is not backup and/or stored
somewhere else. so that puts me at a -1 on the migration utility. but I
might be wrong above most if using hbase they should be able to output the
data to a flat file in hadoop to be imported later after the update.
Billy
"Jim Kellerman" <ji...@powerset.com> wrote in
message
news:84E2AE771361E9419DD0EFBD31F09C4D4C8C3379F1@EXVMBX015-1.exch015.msoutlookonline.net...
Do you have data stored in HBase that you cannot recreate?
HADOOP-2478 will introduce an incompatible change in how HBase
lays out files in HDFS so that should the root or meta tables
be corrupted, it will be possible to reconstruct them from
information in the file system alone.
The problem is in building a migration utility. Anything that
we could build to migrate from the current file structure to
the new file structure would require that the root an meta
regions be absolutely correct. If they are not, the migration
would fail, because there is not enough information on disk
currently to rebuild the root and meta regions.
Is it acceptable for this change to be made without the provision
of an upgrade utility?
If not, are you willing to accept the risk that the upgrade
may fail if you have corruption in your root or meta regions?
After HADOOP-2478, we will be able to build a fault tolerant
upgrade utility, should HBase's file structure change again.
Additionally, we will be able to provide the equivalent of
fsck for HBase after HADOOP-2478.
---
Jim Kellerman, Senior Engineer; Powerset
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date: 1/2/2008
11:29 AM
Re: Question for HBase users
Posted by Taeho Kang <tk...@gmail.com>.
Hi Lars,
The test result for the effect of optimization can be found at the bottom of
this link http://issues.apache.org/jira/browse/HADOOP-1687
However, If you were using the latest version of Hadoop (0.15 and up), then
all the namenode optimization would've been built in.
Using Zip archive is another approach. If storing data to HDFS is the only
thing you want, this approach should be good enough.
However, if you want some data processing done, you have to keep mind that a
zip file on HDFS cannot be processed with Map Reduce. It first has to be
downloaded to a local filesystem, and then processed there locally (which is
slow and no good!)
If you want to parallel-process the data in a zip file on HDFS with Hadoop
Map Reduce...
(1) download zip to local FS
(2) unzip it
(3) upload the unzipped data to HDFS
(4) run map reduce.
That's quite a bit of work (and it's going to take a lot of time, too....)
Let me know if I could give you more help.
/Taeho
On Jan 7, 2008 1:26 PM, Lars George <la...@worldlingo.com> wrote:
> Hi Taeho,
>
> > Fortunately for us, we don't have a need for storing millions of files
> in
> > HDFS just yet. We are adding only a few thousand files a day, so that
> gives
> > us a handful of days. And we've been using Hadoop more than a year, and
> its
> > reliability has been superb.
> >
>
> Sounds great.
>
> > This is just a rough estimation, but we see that 1GB of RAM is required
> in
> > namenode for every 1 million files. Newer versions of Hadoop have more
> > optimized namenode, hence it could host more files. But to be
> conservative,
> > we see 6-7 million files is the limit for a 8GB namenode machine.
> >
>
> Ah, that would explain why my first attempt failed, I have a namenode
> with 1GB of RAM running. That worked OK up to about 3m files, then it
> died - completely. I am using now a nightly build of Hadoop/Hbase, does
> that mean I am in better shape now? How much better does it perform?
>
> > I don't think adding the "consolidation" feature into Hadoop is a good
> > idea.
> > As I said, you may have to add an "layer" that does the consolidation
> work,
> > and use that layer only when necessary.
> >
>
> Yes of course, that is what I meant, we have to handle the creation of
> the slaps on our end. But that is where I think we have to reinvent the
> wheel so to speak.
>
> > As far as the performance is concerned, I don't think it's much of an
> issue.
> > The only cost I can think of is the time taken to make a query to a DB
> plus
> > some time to find the desired file from a given "slap."
> >
>
> OK, my concern is more the size of each slap. Doing some quick math
> (correct me if I am wrong), 80TB total storage divided by say a max of
> 1m slaps means 83MB per slap. That is quite a chunk to load. Unless I
> can do a positioned load of the chunk out of a slap. Does Hadoop have a
> seek load feature?
>
> > Also, you may also create a slap in a way no one file can overlap more
> than
> > one slap.
> >
>
> Yes, that makes sense. I could think of for example simply add files
> together, like an mbox. Or use a ZIP archive. First I would cache enough
> files in a scratch directory in Hadoop and then archive them as one
> slap. (Again that sounds similar to what Hbase is doing?)
>
> > Updates... woo.. here we go again. Hadoop is not designed to handle this
> > need. Basically, its HDFS is designed for large files that rarely change
> -
> >
>
> Yes, understood. I could think of replacing whole slaps, or delete slaps
> once all contained files are obsolete.
>
> > Let us know how your situation goes.
> >
>
> Will do.
>
> Lars
>
>
--
Taeho Kang [tkang.blogspot.com]
Software Engineer, NHN Corporation, Korea
Re: Question for HBase users
Posted by Lars George <la...@worldlingo.com>.
Hi Taeho,
> Fortunately for us, we don't have a need for storing millions of files in
> HDFS just yet. We are adding only a few thousand files a day, so that gives
> us a handful of days. And we've been using Hadoop more than a year, and its
> reliability has been superb.
>
Sounds great.
> This is just a rough estimation, but we see that 1GB of RAM is required in
> namenode for every 1 million files. Newer versions of Hadoop have more
> optimized namenode, hence it could host more files. But to be conservative,
> we see 6-7 million files is the limit for a 8GB namenode machine.
>
Ah, that would explain why my first attempt failed, I have a namenode
with 1GB of RAM running. That worked OK up to about 3m files, then it
died - completely. I am using now a nightly build of Hadoop/Hbase, does
that mean I am in better shape now? How much better does it perform?
> I don't think adding the "consolidation" feature into Hadoop is a good
> idea.
> As I said, you may have to add an "layer" that does the consolidation work,
> and use that layer only when necessary.
>
Yes of course, that is what I meant, we have to handle the creation of
the slaps on our end. But that is where I think we have to reinvent the
wheel so to speak.
> As far as the performance is concerned, I don't think it's much of an issue.
> The only cost I can think of is the time taken to make a query to a DB plus
> some time to find the desired file from a given "slap."
>
OK, my concern is more the size of each slap. Doing some quick math
(correct me if I am wrong), 80TB total storage divided by say a max of
1m slaps means 83MB per slap. That is quite a chunk to load. Unless I
can do a positioned load of the chunk out of a slap. Does Hadoop have a
seek load feature?
> Also, you may also create a slap in a way no one file can overlap more than
> one slap.
>
Yes, that makes sense. I could think of for example simply add files
together, like an mbox. Or use a ZIP archive. First I would cache enough
files in a scratch directory in Hadoop and then archive them as one
slap. (Again that sounds similar to what Hbase is doing?)
> Updates... woo.. here we go again. Hadoop is not designed to handle this
> need. Basically, its HDFS is designed for large files that rarely change -
>
Yes, understood. I could think of replacing whole slaps, or delete slaps
once all contained files are obsolete.
> Let us know how your situation goes.
>
Will do.
Lars
Re: Question for HBase users
Posted by Taeho Kang <tk...@gmail.com>.
Hello, Lars.
> Thank you for the input. May I ask where your experience comes from?
> What was the largest you have seen in real live? Just asking.
Fortunately for us, we don't have a need for storing millions of files in
HDFS just yet. We are adding only a few thousand files a day, so that gives
us a handful of days. And we've been using Hadoop more than a year, and its
reliability has been superb.
This is just a rough estimation, but we see that 1GB of RAM is required in
namenode for every 1 million files. Newer versions of Hadoop have more
optimized namenode, hence it could host more files. But to be conservative,
we see 6-7 million files is the limit for a 8GB namenode machine.
As far as consolidating files is concerned, this sound like having to
> store small databases in Hadoop. But then how is that for performance
> when needing for example two small files in succession but both are from
> a different slap, then I would have to stream in the slap and then read
> the small chunk while dropping the slap just afterwards. Or is it
> possible to specify an offset during the streaming of a file, hence only
> loading the chunk out of the slap?
I don't think adding the "consolidation" feature into Hadoop is a good idea.
As I said, you may have to add an "layer" that does the consolidation work,
and use that layer only when necessary.
As far as the performance is concerned, I don't think it's much of an issue.
The only cost I can think of is the time taken to make a query to a DB plus
some time to find the desired file from a given "slap."
Also, you may also create a slap in a way no one file can overlap more than
one slap.
And then how to I handle - although rare - updates of small files? I
> would have to remember where the new file is (or that the old is
> outdated etc.) meaning I have to track locations and save those too
> somewhere reliably. Isn't that what Hbase promises to do in essence?
Updates... woo.. here we go again. Hadoop is not designed to handle this
need. Basically, its HDFS is designed for large files that rarely change -
no appending, no update in the middle. HBase needs update/append features to
do what it wants to do, but I haven't really had much experience with it to
give you a comment on how well it works.
Let us know how your situation goes.
> Taeho Kang wrote:
> > Thanks for sharing your "painful" experience with us, Lars. I always
> > wondered what would hapeen if HDFS tried to host few hundred million
> files.
> >
> > By the way, I think, with the current namenode desgin of Hadoop, it is
> > unlikely that you will ever be able to host 500 million files on a
> single
> > cluster unless you have a few hundred GB of RAM in a single machine.
> >
> > For your problems, you might want to decrease number of files
> dramatically
> > - into N million files, where 0 < N < 10~20, to be safe. The N really
> > depends on how much RAM you have in your namenode.
> >
> > For more info on how one may calculate the max number of files in
> > HDFS, you may want to take a look at
> > http://issues.apache.org/jira/browse/HADOOP-1687.
> >
> > You might also want to have some "layer" that combines lots of small
> files
> > into one big file, and that one big file is the one that gets into HDFS.
> >
> > Let us know how your project goes... :-)
> >
> > On 1/6/08, Lars George <la...@worldlingo.com> wrote:
> >
> >> Ted,
> >>
> >> In an absolute worst case scenario. I know this is beta and all, but I
> >> start using HBase in a production environment and need to limit
> downtime
> >> (which is what this architecture promises) to minimum - none at all if
> I
> >> can.
> >>
> >> All in all, if I cannot rely on HBase yet being stable, what would your
> >> (or do others) recommend to store say 500m documents on a set of 40
> >> servers with about 1.7TB each. I need this to be reliable, fail-over
> >> safe etc. Basically the Hadoop approach, but without it. And storage is
> >> only one question, how do I parallelize the computing to make use of 80
> >> CPU's?
> >>
> >> Initially I used Hadoop directly and what happened is that after 4m
> >> documents the HDFS simply died on me, the name server would not start
> >> anymore and I had to format the whole thing again and insert the files
> >> once more, this time in HBase.
> >>
> >> I assume it was my rather simplistic directory structure, but I was not
> >> able to find any FAQ or hint of whatever kind how to lay out the files
> >> best in Hadoop, I had to go with the first assumption. If I would know
> >> how I can use HDFS reliable, then I would not worry too much.
> >>
> >> Is there anyone out there that could help me getting this sorted out? I
> >> am willing to pay consulting fees if I have to. At the moment I am at a
> >> loss - sure I trial and error approach would keep me going forward, but
> >> I am on a tight deadline too and that counters that approach.
> >>
> >> Any help is appreciated.
> >>
> >> Thanks,
> >> Lars
> >>
> >>
> >> Ted Dunning wrote:
> >>
> >>> Lars,
> >>>
> >>> Can you dump your documents to external storage (either HDFS or
> ordinary
> >>> file space storage)?
> >>>
> >>>
> >>> On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
> >>>
> >>>
> >>>
> >>>> Jim,
> >>>>
> >>>> I have inserted about 5million documents into HBase and translate
> them
> >>>>
> >> into
> >>
> >>>> 15 languages (means I end up with about 75million in the end). That
> >>>>
> >> data is
> >>
> >>>> only recreatable if we process them costly again. So I am in need of
> a
> >>>> migration path.
> >>>>
> >>>> For me this is a definitely +1 for a migration tool.
> >>>>
> >>>> Sorry to be a hassle like this. :\
> >>>>
> >>>> Lars
> >>>>
> >>>> ----
> >>>> Lars George, CTO
> >>>> WorldLingo
> >>>>
> >>>>
> >>>> Jim Kellerman wrote:
> >>>>
> >>>>
> >>>>> Do you have data stored in HBase that you cannot recreate?
> >>>>>
> >>>>> HADOOP-2478 will introduce an incompatible change in how HBase
> >>>>> lays out files in HDFS so that should the root or meta tables
> >>>>> be corrupted, it will be possible to reconstruct them from
> >>>>> information in the file system alone.
> >>>>>
> >>>>> The problem is in building a migration utility. Anything that
> >>>>> we could build to migrate from the current file structure to
> >>>>> the new file structure would require that the root an meta
> >>>>> regions be absolutely correct. If they are not, the migration
> >>>>> would fail, because there is not enough information on disk
> >>>>> currently to rebuild the root and meta regions.
> >>>>>
> >>>>> Is it acceptable for this change to be made without the provision
> >>>>> of an upgrade utility?
> >>>>>
> >>>>> If not, are you willing to accept the risk that the upgrade
> >>>>> may fail if you have corruption in your root or meta regions?
> >>>>>
> >>>>> After HADOOP-2478, we will be able to build a fault tolerant
> >>>>> upgrade utility, should HBase's file structure change again.
> >>>>> Additionally, we will be able to provide the equivalent of
> >>>>> fsck for HBase after HADOOP-2478.
> >>>>>
> >>>>> ---
> >>>>> Jim Kellerman, Senior Engineer; Powerset
> >>>>>
> >>>>> No virus found in this outgoing message.
> >>>>> Checked by AVG Free Edition.
> >>>>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
> >>>>>
> >> 1/2/2008
> >>
> >>>>> 11:29 AM
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>
> >
> >
> >
> >
>
--
Taeho Kang [tkang.blogspot.com]
Software Engineer, NHN Corporation, Korea
Re: Question for HBase users
Posted by Lars George <la...@worldlingo.com>.
Hi Taeho,
Thank you for the input. May I ask where your experience comes from?
What was the largest you have seen in real live? Just asking.
As far as consolidating files is concerned, this sound like having to
store small databases in Hadoop. But then how is that for performance
when needing for example two small files in succession but both are from
a different slap, then I would have to stream in the slap and then read
the small chunk while dropping the slap just afterwards. Or is it
possible to specify an offset during the streaming of a file, hence only
loading the chunk out of the slap?
And then how to I handle - although rare - updates of small files? I
would have to remember where the new file is (or that the old is
outdated etc.) meaning I have to track locations and save those too
somewhere reliably. Isn't that what Hbase promises to do in essence?
Well, once again, I am grateful for every hint I can get.
Lars
Taeho Kang wrote:
> Thanks for sharing your "painful" experience with us, Lars. I always
> wondered what would hapeen if HDFS tried to host few hundred million files.
>
> By the way, I think, with the current namenode desgin of Hadoop, it is
> unlikely that you will ever be able to host 500 million files on a single
> cluster unless you have a few hundred GB of RAM in a single machine.
>
> For your problems, you might want to decrease number of files dramatically
> - into N million files, where 0 < N < 10~20, to be safe. The N really
> depends on how much RAM you have in your namenode.
>
> For more info on how one may calculate the max number of files in
> HDFS, you may want to take a look at
> http://issues.apache.org/jira/browse/HADOOP-1687.
>
> You might also want to have some "layer" that combines lots of small files
> into one big file, and that one big file is the one that gets into HDFS.
>
> Let us know how your project goes... :-)
>
> On 1/6/08, Lars George <la...@worldlingo.com> wrote:
>
>> Ted,
>>
>> In an absolute worst case scenario. I know this is beta and all, but I
>> start using HBase in a production environment and need to limit downtime
>> (which is what this architecture promises) to minimum - none at all if I
>> can.
>>
>> All in all, if I cannot rely on HBase yet being stable, what would your
>> (or do others) recommend to store say 500m documents on a set of 40
>> servers with about 1.7TB each. I need this to be reliable, fail-over
>> safe etc. Basically the Hadoop approach, but without it. And storage is
>> only one question, how do I parallelize the computing to make use of 80
>> CPU's?
>>
>> Initially I used Hadoop directly and what happened is that after 4m
>> documents the HDFS simply died on me, the name server would not start
>> anymore and I had to format the whole thing again and insert the files
>> once more, this time in HBase.
>>
>> I assume it was my rather simplistic directory structure, but I was not
>> able to find any FAQ or hint of whatever kind how to lay out the files
>> best in Hadoop, I had to go with the first assumption. If I would know
>> how I can use HDFS reliable, then I would not worry too much.
>>
>> Is there anyone out there that could help me getting this sorted out? I
>> am willing to pay consulting fees if I have to. At the moment I am at a
>> loss - sure I trial and error approach would keep me going forward, but
>> I am on a tight deadline too and that counters that approach.
>>
>> Any help is appreciated.
>>
>> Thanks,
>> Lars
>>
>>
>> Ted Dunning wrote:
>>
>>> Lars,
>>>
>>> Can you dump your documents to external storage (either HDFS or ordinary
>>> file space storage)?
>>>
>>>
>>> On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
>>>
>>>
>>>
>>>> Jim,
>>>>
>>>> I have inserted about 5million documents into HBase and translate them
>>>>
>> into
>>
>>>> 15 languages (means I end up with about 75million in the end). That
>>>>
>> data is
>>
>>>> only recreatable if we process them costly again. So I am in need of a
>>>> migration path.
>>>>
>>>> For me this is a definitely +1 for a migration tool.
>>>>
>>>> Sorry to be a hassle like this. :\
>>>>
>>>> Lars
>>>>
>>>> ----
>>>> Lars George, CTO
>>>> WorldLingo
>>>>
>>>>
>>>> Jim Kellerman wrote:
>>>>
>>>>
>>>>> Do you have data stored in HBase that you cannot recreate?
>>>>>
>>>>> HADOOP-2478 will introduce an incompatible change in how HBase
>>>>> lays out files in HDFS so that should the root or meta tables
>>>>> be corrupted, it will be possible to reconstruct them from
>>>>> information in the file system alone.
>>>>>
>>>>> The problem is in building a migration utility. Anything that
>>>>> we could build to migrate from the current file structure to
>>>>> the new file structure would require that the root an meta
>>>>> regions be absolutely correct. If they are not, the migration
>>>>> would fail, because there is not enough information on disk
>>>>> currently to rebuild the root and meta regions.
>>>>>
>>>>> Is it acceptable for this change to be made without the provision
>>>>> of an upgrade utility?
>>>>>
>>>>> If not, are you willing to accept the risk that the upgrade
>>>>> may fail if you have corruption in your root or meta regions?
>>>>>
>>>>> After HADOOP-2478, we will be able to build a fault tolerant
>>>>> upgrade utility, should HBase's file structure change again.
>>>>> Additionally, we will be able to provide the equivalent of
>>>>> fsck for HBase after HADOOP-2478.
>>>>>
>>>>> ---
>>>>> Jim Kellerman, Senior Engineer; Powerset
>>>>>
>>>>> No virus found in this outgoing message.
>>>>> Checked by AVG Free Edition.
>>>>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
>>>>>
>> 1/2/2008
>>
>>>>> 11:29 AM
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>
>
>
>
Re: Question for HBase users
Posted by Lars George <la...@worldlingo.com>.
Ted,
This is actually both, I first tried Hadoop directly and then HBase in
my second attempt.
Lars
Ted Dunning wrote:
> He didn't store all of these documents in separate files. He stored them in
> hbase, hence his pain with the upgrade.
>
>
> On 1/6/08 5:44 PM, "Taeho Kang" <tk...@gmail.com> wrote:
>
>
>> Thanks for sharing your "painful" experience with us, Lars. I always
>> wondered what would hapeen if HDFS tried to host few hundred million files.
>>
>> By the way, I think, with the current namenode desgin of Hadoop, it is
>> unlikely that you will ever be able to host 500 million files on a single
>> cluster unless you have a few hundred GB of RAM in a single machine.
>>
>> For your problems, you might want to decrease number of files dramatically
>> - into N million files, where 0 < N < 10~20, to be safe. The N really
>> depends on how much RAM you have in your namenode.
>>
>> For more info on how one may calculate the max number of files in
>> HDFS, you may want to take a look at
>> http://issues.apache.org/jira/browse/HADOOP-1687.
>>
>> You might also want to have some "layer" that combines lots of small files
>> into one big file, and that one big file is the one that gets into HDFS.
>>
>> Let us know how your project goes... :-)
>>
>> On 1/6/08, Lars George <la...@worldlingo.com> wrote:
>>
>>> Ted,
>>>
>>> In an absolute worst case scenario. I know this is beta and all, but I
>>> start using HBase in a production environment and need to limit downtime
>>> (which is what this architecture promises) to minimum - none at all if I
>>> can.
>>>
>>> All in all, if I cannot rely on HBase yet being stable, what would your
>>> (or do others) recommend to store say 500m documents on a set of 40
>>> servers with about 1.7TB each. I need this to be reliable, fail-over
>>> safe etc. Basically the Hadoop approach, but without it. And storage is
>>> only one question, how do I parallelize the computing to make use of 80
>>> CPU's?
>>>
>>> Initially I used Hadoop directly and what happened is that after 4m
>>> documents the HDFS simply died on me, the name server would not start
>>> anymore and I had to format the whole thing again and insert the files
>>> once more, this time in HBase.
>>>
>>> I assume it was my rather simplistic directory structure, but I was not
>>> able to find any FAQ or hint of whatever kind how to lay out the files
>>> best in Hadoop, I had to go with the first assumption. If I would know
>>> how I can use HDFS reliable, then I would not worry too much.
>>>
>>> Is there anyone out there that could help me getting this sorted out? I
>>> am willing to pay consulting fees if I have to. At the moment I am at a
>>> loss - sure I trial and error approach would keep me going forward, but
>>> I am on a tight deadline too and that counters that approach.
>>>
>>> Any help is appreciated.
>>>
>>> Thanks,
>>> Lars
>>>
>>>
>>> Ted Dunning wrote:
>>>
>>>> Lars,
>>>>
>>>> Can you dump your documents to external storage (either HDFS or ordinary
>>>> file space storage)?
>>>>
>>>>
>>>> On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
>>>>
>>>>
>>>>
>>>>> Jim,
>>>>>
>>>>> I have inserted about 5million documents into HBase and translate them
>>>>>
>>> into
>>>
>>>>> 15 languages (means I end up with about 75million in the end). That
>>>>>
>>> data is
>>>
>>>>> only recreatable if we process them costly again. So I am in need of a
>>>>> migration path.
>>>>>
>>>>> For me this is a definitely +1 for a migration tool.
>>>>>
>>>>> Sorry to be a hassle like this. :\
>>>>>
>>>>> Lars
>>>>>
>>>>> ----
>>>>> Lars George, CTO
>>>>> WorldLingo
>>>>>
>>>>>
>>>>> Jim Kellerman wrote:
>>>>>
>>>>>
>>>>>> Do you have data stored in HBase that you cannot recreate?
>>>>>>
>>>>>> HADOOP-2478 will introduce an incompatible change in how HBase
>>>>>> lays out files in HDFS so that should the root or meta tables
>>>>>> be corrupted, it will be possible to reconstruct them from
>>>>>> information in the file system alone.
>>>>>>
>>>>>> The problem is in building a migration utility. Anything that
>>>>>> we could build to migrate from the current file structure to
>>>>>> the new file structure would require that the root an meta
>>>>>> regions be absolutely correct. If they are not, the migration
>>>>>> would fail, because there is not enough information on disk
>>>>>> currently to rebuild the root and meta regions.
>>>>>>
>>>>>> Is it acceptable for this change to be made without the provision
>>>>>> of an upgrade utility?
>>>>>>
>>>>>> If not, are you willing to accept the risk that the upgrade
>>>>>> may fail if you have corruption in your root or meta regions?
>>>>>>
>>>>>> After HADOOP-2478, we will be able to build a fault tolerant
>>>>>> upgrade utility, should HBase's file structure change again.
>>>>>> Additionally, we will be able to provide the equivalent of
>>>>>> fsck for HBase after HADOOP-2478.
>>>>>>
>>>>>> ---
>>>>>> Jim Kellerman, Senior Engineer; Powerset
>>>>>>
>>>>>> No virus found in this outgoing message.
>>>>>> Checked by AVG Free Edition.
>>>>>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
>>>>>>
>>> 1/2/2008
>>>
>>>>>> 11:29 AM
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>
>
Re: Question for HBase users
Posted by Ted Dunning <td...@veoh.com>.
He didn't store all of these documents in separate files. He stored them in
hbase, hence his pain with the upgrade.
On 1/6/08 5:44 PM, "Taeho Kang" <tk...@gmail.com> wrote:
> Thanks for sharing your "painful" experience with us, Lars. I always
> wondered what would hapeen if HDFS tried to host few hundred million files.
>
> By the way, I think, with the current namenode desgin of Hadoop, it is
> unlikely that you will ever be able to host 500 million files on a single
> cluster unless you have a few hundred GB of RAM in a single machine.
>
> For your problems, you might want to decrease number of files dramatically
> - into N million files, where 0 < N < 10~20, to be safe. The N really
> depends on how much RAM you have in your namenode.
>
> For more info on how one may calculate the max number of files in
> HDFS, you may want to take a look at
> http://issues.apache.org/jira/browse/HADOOP-1687.
>
> You might also want to have some "layer" that combines lots of small files
> into one big file, and that one big file is the one that gets into HDFS.
>
> Let us know how your project goes... :-)
>
> On 1/6/08, Lars George <la...@worldlingo.com> wrote:
>>
>> Ted,
>>
>> In an absolute worst case scenario. I know this is beta and all, but I
>> start using HBase in a production environment and need to limit downtime
>> (which is what this architecture promises) to minimum - none at all if I
>> can.
>>
>> All in all, if I cannot rely on HBase yet being stable, what would your
>> (or do others) recommend to store say 500m documents on a set of 40
>> servers with about 1.7TB each. I need this to be reliable, fail-over
>> safe etc. Basically the Hadoop approach, but without it. And storage is
>> only one question, how do I parallelize the computing to make use of 80
>> CPU's?
>>
>> Initially I used Hadoop directly and what happened is that after 4m
>> documents the HDFS simply died on me, the name server would not start
>> anymore and I had to format the whole thing again and insert the files
>> once more, this time in HBase.
>>
>> I assume it was my rather simplistic directory structure, but I was not
>> able to find any FAQ or hint of whatever kind how to lay out the files
>> best in Hadoop, I had to go with the first assumption. If I would know
>> how I can use HDFS reliable, then I would not worry too much.
>>
>> Is there anyone out there that could help me getting this sorted out? I
>> am willing to pay consulting fees if I have to. At the moment I am at a
>> loss - sure I trial and error approach would keep me going forward, but
>> I am on a tight deadline too and that counters that approach.
>>
>> Any help is appreciated.
>>
>> Thanks,
>> Lars
>>
>>
>> Ted Dunning wrote:
>>> Lars,
>>>
>>> Can you dump your documents to external storage (either HDFS or ordinary
>>> file space storage)?
>>>
>>>
>>> On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
>>>
>>>
>>>> Jim,
>>>>
>>>> I have inserted about 5million documents into HBase and translate them
>> into
>>>> 15 languages (means I end up with about 75million in the end). That
>> data is
>>>> only recreatable if we process them costly again. So I am in need of a
>>>> migration path.
>>>>
>>>> For me this is a definitely +1 for a migration tool.
>>>>
>>>> Sorry to be a hassle like this. :\
>>>>
>>>> Lars
>>>>
>>>> ----
>>>> Lars George, CTO
>>>> WorldLingo
>>>>
>>>>
>>>> Jim Kellerman wrote:
>>>>
>>>>> Do you have data stored in HBase that you cannot recreate?
>>>>>
>>>>> HADOOP-2478 will introduce an incompatible change in how HBase
>>>>> lays out files in HDFS so that should the root or meta tables
>>>>> be corrupted, it will be possible to reconstruct them from
>>>>> information in the file system alone.
>>>>>
>>>>> The problem is in building a migration utility. Anything that
>>>>> we could build to migrate from the current file structure to
>>>>> the new file structure would require that the root an meta
>>>>> regions be absolutely correct. If they are not, the migration
>>>>> would fail, because there is not enough information on disk
>>>>> currently to rebuild the root and meta regions.
>>>>>
>>>>> Is it acceptable for this change to be made without the provision
>>>>> of an upgrade utility?
>>>>>
>>>>> If not, are you willing to accept the risk that the upgrade
>>>>> may fail if you have corruption in your root or meta regions?
>>>>>
>>>>> After HADOOP-2478, we will be able to build a fault tolerant
>>>>> upgrade utility, should HBase's file structure change again.
>>>>> Additionally, we will be able to provide the equivalent of
>>>>> fsck for HBase after HADOOP-2478.
>>>>>
>>>>> ---
>>>>> Jim Kellerman, Senior Engineer; Powerset
>>>>>
>>>>> No virus found in this outgoing message.
>>>>> Checked by AVG Free Edition.
>>>>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
>> 1/2/2008
>>>>> 11:29 AM
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>
>
Re: Question for HBase users
Posted by Taeho Kang <tk...@gmail.com>.
Thanks for sharing your "painful" experience with us, Lars. I always
wondered what would hapeen if HDFS tried to host few hundred million files.
By the way, I think, with the current namenode desgin of Hadoop, it is
unlikely that you will ever be able to host 500 million files on a single
cluster unless you have a few hundred GB of RAM in a single machine.
For your problems, you might want to decrease number of files dramatically
- into N million files, where 0 < N < 10~20, to be safe. The N really
depends on how much RAM you have in your namenode.
For more info on how one may calculate the max number of files in
HDFS, you may want to take a look at
http://issues.apache.org/jira/browse/HADOOP-1687.
You might also want to have some "layer" that combines lots of small files
into one big file, and that one big file is the one that gets into HDFS.
Let us know how your project goes... :-)
On 1/6/08, Lars George <la...@worldlingo.com> wrote:
>
> Ted,
>
> In an absolute worst case scenario. I know this is beta and all, but I
> start using HBase in a production environment and need to limit downtime
> (which is what this architecture promises) to minimum - none at all if I
> can.
>
> All in all, if I cannot rely on HBase yet being stable, what would your
> (or do others) recommend to store say 500m documents on a set of 40
> servers with about 1.7TB each. I need this to be reliable, fail-over
> safe etc. Basically the Hadoop approach, but without it. And storage is
> only one question, how do I parallelize the computing to make use of 80
> CPU's?
>
> Initially I used Hadoop directly and what happened is that after 4m
> documents the HDFS simply died on me, the name server would not start
> anymore and I had to format the whole thing again and insert the files
> once more, this time in HBase.
>
> I assume it was my rather simplistic directory structure, but I was not
> able to find any FAQ or hint of whatever kind how to lay out the files
> best in Hadoop, I had to go with the first assumption. If I would know
> how I can use HDFS reliable, then I would not worry too much.
>
> Is there anyone out there that could help me getting this sorted out? I
> am willing to pay consulting fees if I have to. At the moment I am at a
> loss - sure I trial and error approach would keep me going forward, but
> I am on a tight deadline too and that counters that approach.
>
> Any help is appreciated.
>
> Thanks,
> Lars
>
>
> Ted Dunning wrote:
> > Lars,
> >
> > Can you dump your documents to external storage (either HDFS or ordinary
> > file space storage)?
> >
> >
> > On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
> >
> >
> >> Jim,
> >>
> >> I have inserted about 5million documents into HBase and translate them
> into
> >> 15 languages (means I end up with about 75million in the end). That
> data is
> >> only recreatable if we process them costly again. So I am in need of a
> >> migration path.
> >>
> >> For me this is a definitely +1 for a migration tool.
> >>
> >> Sorry to be a hassle like this. :\
> >>
> >> Lars
> >>
> >> ----
> >> Lars George, CTO
> >> WorldLingo
> >>
> >>
> >> Jim Kellerman wrote:
> >>
> >>> Do you have data stored in HBase that you cannot recreate?
> >>>
> >>> HADOOP-2478 will introduce an incompatible change in how HBase
> >>> lays out files in HDFS so that should the root or meta tables
> >>> be corrupted, it will be possible to reconstruct them from
> >>> information in the file system alone.
> >>>
> >>> The problem is in building a migration utility. Anything that
> >>> we could build to migrate from the current file structure to
> >>> the new file structure would require that the root an meta
> >>> regions be absolutely correct. If they are not, the migration
> >>> would fail, because there is not enough information on disk
> >>> currently to rebuild the root and meta regions.
> >>>
> >>> Is it acceptable for this change to be made without the provision
> >>> of an upgrade utility?
> >>>
> >>> If not, are you willing to accept the risk that the upgrade
> >>> may fail if you have corruption in your root or meta regions?
> >>>
> >>> After HADOOP-2478, we will be able to build a fault tolerant
> >>> upgrade utility, should HBase's file structure change again.
> >>> Additionally, we will be able to provide the equivalent of
> >>> fsck for HBase after HADOOP-2478.
> >>>
> >>> ---
> >>> Jim Kellerman, Senior Engineer; Powerset
> >>>
> >>> No virus found in this outgoing message.
> >>> Checked by AVG Free Edition.
> >>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date:
> 1/2/2008
> >>> 11:29 AM
> >>>
> >>>
> >>>
> >>>
> >
> >
>
--
Taeho Kang [tkang.blogspot.com]
Software Engineer, NHN Corporation, Korea
Re: Question for HBase users
Posted by Lars George <la...@worldlingo.com>.
Ted,
In an absolute worst case scenario. I know this is beta and all, but I
start using HBase in a production environment and need to limit downtime
(which is what this architecture promises) to minimum - none at all if I
can.
All in all, if I cannot rely on HBase yet being stable, what would your
(or do others) recommend to store say 500m documents on a set of 40
servers with about 1.7TB each. I need this to be reliable, fail-over
safe etc. Basically the Hadoop approach, but without it. And storage is
only one question, how do I parallelize the computing to make use of 80
CPU's?
Initially I used Hadoop directly and what happened is that after 4m
documents the HDFS simply died on me, the name server would not start
anymore and I had to format the whole thing again and insert the files
once more, this time in HBase.
I assume it was my rather simplistic directory structure, but I was not
able to find any FAQ or hint of whatever kind how to lay out the files
best in Hadoop, I had to go with the first assumption. If I would know
how I can use HDFS reliable, then I would not worry too much.
Is there anyone out there that could help me getting this sorted out? I
am willing to pay consulting fees if I have to. At the moment I am at a
loss - sure I trial and error approach would keep me going forward, but
I am on a tight deadline too and that counters that approach.
Any help is appreciated.
Thanks,
Lars
Ted Dunning wrote:
> Lars,
>
> Can you dump your documents to external storage (either HDFS or ordinary
> file space storage)?
>
>
> On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
>
>
>> Jim,
>>
>> I have inserted about 5million documents into HBase and translate them into
>> 15 languages (means I end up with about 75million in the end). That data is
>> only recreatable if we process them costly again. So I am in need of a
>> migration path.
>>
>> For me this is a definitely +1 for a migration tool.
>>
>> Sorry to be a hassle like this. :\
>>
>> Lars
>>
>> ----
>> Lars George, CTO
>> WorldLingo
>>
>>
>> Jim Kellerman wrote:
>>
>>> Do you have data stored in HBase that you cannot recreate?
>>>
>>> HADOOP-2478 will introduce an incompatible change in how HBase
>>> lays out files in HDFS so that should the root or meta tables
>>> be corrupted, it will be possible to reconstruct them from
>>> information in the file system alone.
>>>
>>> The problem is in building a migration utility. Anything that
>>> we could build to migrate from the current file structure to
>>> the new file structure would require that the root an meta
>>> regions be absolutely correct. If they are not, the migration
>>> would fail, because there is not enough information on disk
>>> currently to rebuild the root and meta regions.
>>>
>>> Is it acceptable for this change to be made without the provision
>>> of an upgrade utility?
>>>
>>> If not, are you willing to accept the risk that the upgrade
>>> may fail if you have corruption in your root or meta regions?
>>>
>>> After HADOOP-2478, we will be able to build a fault tolerant
>>> upgrade utility, should HBase's file structure change again.
>>> Additionally, we will be able to provide the equivalent of
>>> fsck for HBase after HADOOP-2478.
>>>
>>> ---
>>> Jim Kellerman, Senior Engineer; Powerset
>>>
>>> No virus found in this outgoing message.
>>> Checked by AVG Free Edition.
>>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date: 1/2/2008
>>> 11:29 AM
>>>
>>>
>>>
>>>
>
>
Re: Question for HBase users
Posted by Ted Dunning <td...@veoh.com>.
Lars,
Can you dump your documents to external storage (either HDFS or ordinary
file space storage)?
On 1/4/08 10:01 PM, "larsgeorge" <la...@worldlingo.com> wrote:
>
> Jim,
>
> I have inserted about 5million documents into HBase and translate them into
> 15 languages (means I end up with about 75million in the end). That data is
> only recreatable if we process them costly again. So I am in need of a
> migration path.
>
> For me this is a definitely +1 for a migration tool.
>
> Sorry to be a hassle like this. :\
>
> Lars
>
> ----
> Lars George, CTO
> WorldLingo
>
>
> Jim Kellerman wrote:
>>
>> Do you have data stored in HBase that you cannot recreate?
>>
>> HADOOP-2478 will introduce an incompatible change in how HBase
>> lays out files in HDFS so that should the root or meta tables
>> be corrupted, it will be possible to reconstruct them from
>> information in the file system alone.
>>
>> The problem is in building a migration utility. Anything that
>> we could build to migrate from the current file structure to
>> the new file structure would require that the root an meta
>> regions be absolutely correct. If they are not, the migration
>> would fail, because there is not enough information on disk
>> currently to rebuild the root and meta regions.
>>
>> Is it acceptable for this change to be made without the provision
>> of an upgrade utility?
>>
>> If not, are you willing to accept the risk that the upgrade
>> may fail if you have corruption in your root or meta regions?
>>
>> After HADOOP-2478, we will be able to build a fault tolerant
>> upgrade utility, should HBase's file structure change again.
>> Additionally, we will be able to provide the equivalent of
>> fsck for HBase after HADOOP-2478.
>>
>> ---
>> Jim Kellerman, Senior Engineer; Powerset
>>
>> No virus found in this outgoing message.
>> Checked by AVG Free Edition.
>> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date: 1/2/2008
>> 11:29 AM
>>
>>
>>
Re: Question for HBase users
Posted by larsgeorge <la...@worldlingo.com>.
Jim,
I have inserted about 5million documents into HBase and translate them into
15 languages (means I end up with about 75million in the end). That data is
only recreatable if we process them costly again. So I am in need of a
migration path.
For me this is a definitely +1 for a migration tool.
Sorry to be a hassle like this. :\
Lars
----
Lars George, CTO
WorldLingo
Jim Kellerman wrote:
>
> Do you have data stored in HBase that you cannot recreate?
>
> HADOOP-2478 will introduce an incompatible change in how HBase
> lays out files in HDFS so that should the root or meta tables
> be corrupted, it will be possible to reconstruct them from
> information in the file system alone.
>
> The problem is in building a migration utility. Anything that
> we could build to migrate from the current file structure to
> the new file structure would require that the root an meta
> regions be absolutely correct. If they are not, the migration
> would fail, because there is not enough information on disk
> currently to rebuild the root and meta regions.
>
> Is it acceptable for this change to be made without the provision
> of an upgrade utility?
>
> If not, are you willing to accept the risk that the upgrade
> may fail if you have corruption in your root or meta regions?
>
> After HADOOP-2478, we will be able to build a fault tolerant
> upgrade utility, should HBase's file structure change again.
> Additionally, we will be able to provide the equivalent of
> fsck for HBase after HADOOP-2478.
>
> ---
> Jim Kellerman, Senior Engineer; Powerset
>
> No virus found in this outgoing message.
> Checked by AVG Free Edition.
> Version: 7.5.516 / Virus Database: 269.17.13/1207 - Release Date: 1/2/2008
> 11:29 AM
>
>
>
--
View this message in context: http://www.nabble.com/Question-for-HBase-users-tp14607732p14630344.html
Sent from the Hadoop Users mailing list archive at Nabble.com.