You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Aaron Eng <ae...@maprtech.com> on 2011/07/01 01:18:02 UTC

Re: Hadoop Java Versions

>Keeping the amount of disks per node low and the amount of nodes high
should keep the impact of dead nodes in control.

It keeps the impact of dead nodes in control but I don't think thats
long-term cost efficient.  As prices of 10GbE go down, the "keep the node
small" arguement seems less fitting.  And on another note, most servers
manufactured in the last 10 years have dual 1GbE network interfaces.  If one
were to go by these calcs:

>150 nodes with four 2TB disks each, with HDFS 60% full, it takes around ~32
minutes to recover

It seems like that assumes a single 1GbE interface, why  not leverage the
second?

On Thu, Jun 30, 2011 at 2:31 PM, Evert Lammerts <Ev...@sara.nl>wrote:

> > You can get 12-24 TB in a server today, which means the loss of a server
> > generates a lot of traffic -which argues for 10 Gbe.
> >
> > But
> >   -big increase in switch cost, especially if you (CoI warning) go with
> > Cisco
> >   -there have been problems with things like BIOS PXE and lights out
> > management on 10 Gbe -probably due to the NICs being things the BIOS
> > wasn't expecting and off the mainboard. This should improve.
> >   -I don't know how well linux works with ether that fast (field reports
> > useful)
> >   -the big threat is still ToR switch failure, as that will trigger a
> > re-replication of every block in the rack.
>
> Keeping the amount of disks per node low and the amount of nodes high
> should keep the impact of dead nodes in control. A ToR switch failing is
> different - missing 30 nodes (~120TB) at once cannot be fixed by adding more
> nodes; that actually increases ToR switch failure. Although such failure is
> quite rare to begin with, I guess. The back-of-the-envelope-calculation I
> made suggests that ~150 (1U) nodes should be fine with 1Gb ethernet. (e.g.,
> when 6 nodes fail in a cluster with 150 nodes with four 2TB disks each, with
> HDFS 60% full, it takes around ~32 minutes to recover. 2 nodes failing
> should take around 640 seconds. Also see the attached spreadsheet.) This
> doesn't take ToR switch failure in account though. On the other hand - 150
> nodes is only ~5 racks - in such a scenario you might rather want to shut
> the system down completely rather than letting it replicate 20% of all data.
>
> Cheers,
> Evert

Re: Hadoop Java Versions

Posted by Ted Dunning <td...@maprtech.com>.
On Fri, Jul 1, 2011 at 9:12 AM, Steve Loughran <st...@apache.org> wrote:

> On 01/07/2011 01:16, Ted Dunning wrote:
>
>> You have to consider the long-term reliability as well.
>>
>> Losing an entire set of 10 or 12 disks at once makes the overall
>> reliability
>> of a large cluster very suspect.  This is because it becomes entirely too
>> likely that two additional drives will fail before the data on the
>> off-line
>> node can be replicated.  For 100 nodes, that can decrease the average time
>> to data loss down to less than a year.
>>
>
> There's also Rodrigo's work on alternate block placement that doesn't
> scatter blocks quite so randomly across a cluster, so a loss of a node or
> rack doesn't have adverse effects on so many files
>
> https://issues.apache.org/**jira/browse/HDFS-1094<https://issues.apache.org/jira/browse/HDFS-1094>


I did calculations based on this as well.  The heuristic level of the
computation is pretty simple, but to go any deeper, you have a pretty hair
computation.  My own approach was to use Monte Carlo Markov Chain to sample
from the failure mode distribution.  The codes that I wrote for this used
pluggable placement, replication and failure models.

I may have lacked sufficient cleverness at the time, but it was very
difficult to come up with structured placement policies that actually
improved the failure probabilities.  Most such strategies massively
decreased average probabilities.  My suspicion by analogy with large
structured error correction codes is that there are structured placement
policies that perform well, but that in reasonably large clusters (number of
disks > 50, say), that random placement will be within epsilon of the best
possible strategy with very high probability.



> Given that most HDDs failures happen on cluster reboot, it is possible for
> 10-12 disks not to come up at the same time, if the cluster has been up for
> a while, but like Todd says -worry. At least a bit.
>

Indeed.

Thank goodness also that disk manufacturers tend to be pessimistic in
quoting MTBF.

These possibilities of correlated failure seriously complicate these
computations, of course.


> I've heard hints of one FS that actually includes HDD batch data in block
> placement, to try and scatter data across batches, and be biased towards
> using new HDDs for temp storage during burn-in. Some research work on doing
> that to HDFS could be something to keep some postgraduate busy for a while,
> "Disk batch-aware block placement".
>

Sadly, I can't comment on my knowledge of this except to say that there are
non-obvious solutions to this that are embedded in at least one commercial
map-reduce related product.  I can't say which without getting chastised.


>  This can only be mitigated in stock
>> hadoop by keeping the number of drives relatively low.
>>
>
> now I'm confused. Do you mean #of HDDs/server, or HDDs/filesystem?


Per system.


> ..with the most number of non-RAIDed HDDs out there -things like Lustre and
> IBM GPFS go for RAID, as does HP IBRIX (the last two of which have some form
> of Hadoop support too, if you ask nicely). HDD/server numbers matter in that
> in a small cluster, it's better to have fewer machines to get more servers
> to spread the data over; you don't really want your 100 TB in three 1U
> servers. As your cluster grows -and you care more about storage capacity
> than raw compute- then the appeal of 24+ TB/server starts to look good, and
> that's when you care about the improvements to datanodes handling loss of
> worker disk better. Even without that, rebooting the DN may fix things, but
> the impact on ongoing work is the big issue -you don't just lose a
> replicated block, you lose data.
>

Generally, I agree with what you say.  The effect of RAID is to squeeze the
error distributions around so that partial failures have lower probability.
  This is complex in the aggregate.


>
> Cascade failures leading to cluster outages are a separate issue and
> normally triggered by switch failure/config than anything else. It doesn't
> matter how reliable the hardware is if it gets the wrong configuration
>
>
Indeed.

Re: Hadoop Java Versions

Posted by Steve Loughran <st...@apache.org>.
On 01/07/2011 01:16, Ted Dunning wrote:
> You have to consider the long-term reliability as well.
>
> Losing an entire set of 10 or 12 disks at once makes the overall reliability
> of a large cluster very suspect.  This is because it becomes entirely too
> likely that two additional drives will fail before the data on the off-line
> node can be replicated.  For 100 nodes, that can decrease the average time
> to data loss down to less than a year.

There's also Rodrigo's work on alternate block placement that doesn't 
scatter blocks quite so randomly across a cluster, so a loss of a node 
or rack doesn't have adverse effects on so many files

https://issues.apache.org/jira/browse/HDFS-1094

Given that most HDDs failures happen on cluster reboot, it is possible 
for 10-12 disks not to come up at the same time, if the cluster has been 
up for a while, but like Todd says -worry. At least a bit.


I've heard hints of one FS that actually includes HDD batch data in 
block placement, to try and scatter data across batches, and be biased 
towards using new HDDs for temp storage during burn-in. Some research 
work on doing that to HDFS could be something to keep some postgraduate 
busy for a while, "Disk batch-aware block placement".

> This can only be mitigated in stock
> hadoop by keeping the number of drives relatively low.

now I'm confused. Do you mean #of HDDs/server, or HDDs/filesystem? 
Because it seems to me that "stock" HDFS's use in production makes it 
one of the filesystems in the planet with the most number of non-RAIDed 
HDDs out there -things like Lustre and IBM GPFS go for RAID, as does HP 
IBRIX (the last two of which have some form of Hadoop support too, if 
you ask nicely). HDD/server numbers matter in that in a small cluster, 
it's better to have fewer machines to get more servers to spread the 
data over; you don't really want your 100 TB in three 1U servers. As 
your cluster grows -and you care more about storage capacity than raw 
compute- then the appeal of 24+ TB/server starts to look good, and 
that's when you care about the improvements to datanodes handling loss 
of worker disk better. Even without that, rebooting the DN may fix 
things, but the impact on ongoing work is the big issue -you don't just 
lose a replicated block, you lose data.

Cascade failures leading to cluster outages are a separate issue and 
normally triggered by switch failure/config than anything else. It 
doesn't matter how reliable the hardware is if it gets the wrong 
configuration


Re: Hadoop Java Versions

Posted by Ted Dunning <td...@maprtech.com>.
Good point Todd.

I was speaking from the experience of people I know who are using 0.20.x

On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon <to...@cloudera.com> wrote:

> On Thu, Jun 30, 2011 at 5:16 PM, Ted Dunning <td...@maprtech.com>
> wrote:
>
> > You have to consider the long-term reliability as well.
> >
> > Losing an entire set of 10 or 12 disks at once makes the overall
> > reliability
> > of a large cluster very suspect.  This is because it becomes entirely too
> > likely that two additional drives will fail before the data on the
> off-line
> > node can be replicated.  For 100 nodes, that can decrease the average
> time
> > to data loss down to less than a year.  This can only be mitigated in
> stock
> > hadoop by keeping the number of drives relatively low.  MapR avoids this
> by
> > not failing nodes for trivial problems.
> >
>
> I'd advise you to look at "stock hadoop" again. This used to be true, but
> was fixed a long while back by HDFS-457 and several followup JIRAs.
>
> If MapR does something fancier, I'm sure we'd be interested to hear about
> it
> so we can compare the approaches.
>

Re: Hadoop Java Versions

Posted by "M. C. Srivas" <mc...@gmail.com>.
No worries.

I read Todd's post as asking for elaboration ... sometimes knowing what
another similar system does helps in improving your own.


On Thu, Jun 30, 2011 at 9:47 PM, Ian Holsman <ha...@holsman.net> wrote:

>
> On Jul 1, 2011, at 2:08 PM, M. C. Srivas wrote:
>
> > On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >>
> >> I'd advise you to look at "stock hadoop" again. This used to be true,
> but
> >> was fixed a long while back by HDFS-457 and several followup JIRAs.
> >>
> >> If MapR does something fancier, I'm sure we'd be interested to hear
> about
> >> it
> >> so we can compare the approaches.
> >>
> >> -Todd
> >>
> >>
> > MapR tracks disk responsiveness. In other words, a moving histogram of
> > IO-completion times is maintained internally, and if a disk starts
> getting
> > really slow, it is pre-emptively taken offline so it does not create long
> > tails for running jobs (and the data on the disk is re-replicated using
> > whatever re-replication policy is in place).  One of the benefits of
> > managing the disks directly instead of through ext3 / xfs / or other ...
> >
> > All these stats can be fed into Ganglia (or pushed out centrally via a
> text
> > file that can be pulled out using NFS)  if historical info about disk
> > behavior (and failures) needs to be preserved.
> >
> > - Srivas.
>
> While I am intrigued about how MapR performs internally, I don't think this
> is the forum for it.
> please keep MapR (and other vendor specific discussions) on their
> respective support forums.
>
> Thanks!
>
> Ian.
>
>

Re: Hadoop Java Versions

Posted by Ted Dunning <td...@maprtech.com>.
Ian,

Good point.  Srivas was responding to Todd's question, but there might be
better fora as you suggest.

We have a good one for specific questions about MapR at
http://answers.mapr.com

That doesn't, however, really provide a useful forum for questions like
Todd's which really spans both domains.  Where would you suggest for
questions that span the two subjects?

On Thu, Jun 30, 2011 at 9:47 PM, Ian Holsman <ha...@holsman.net> wrote:

> >> If MapR does something fancier, I'm sure we'd be interested to hear
> about
> >> it
> >> so we can compare the approaches.
> >>
> >> -Todd
> >>
> >>
> > MapR tracks disk responsiveness. In other words, a moving histogram of
> > IO-completion times is maintained internally, and if a disk starts
> getting
> > really slow, it is pre-emptively taken offline so it does not create long
> > tails for running jobs (and the data on the disk is re-replicated using
> > whatever re-replication policy is in place).  One of the benefits of
> > managing the disks directly instead of through ext3 / xfs / or other ...
> >
> > All these stats can be fed into Ganglia (or pushed out centrally via a
> text
> > file that can be pulled out using NFS)  if historical info about disk
> > behavior (and failures) needs to be preserved.
> >
> > - Srivas.
>
> While I am intrigued about how MapR performs internally, I don't think this
> is the forum for it.
> please keep MapR (and other vendor specific discussions) on their
> respective support forums.
>

Re: Hadoop Java Versions

Posted by Ted Dunning <td...@maprtech.com>.
On Wed, Jul 13, 2011 at 7:59 AM, Eric Baldeschwieler <eric14@hortonworks.com
> wrote:

> We could create an apache hadoop list of product selection discussions.  I
> believe this list is intended to be focused on project governance and
> similar discussions.  Maybe we should simply create a governance list and
> leave this one to be the free for all?
>

If you need to separate the discussions, then taking the discussion with
smaller/more experienced set of participants to the new list is going to be
easier than trying to sweep the tide.  A list named "general" sounds a lot
like "general discussions" to newcomers (and frankly, to old-timers like
me).

For that matter, is there a strong reason to segregate the discussions?

Re: Hadoop Java Versions

Posted by Eric Baldeschwieler <er...@hortonworks.com>.
We could create an apache hadoop list of product selection discussions.  I believe this list is intended to be focused on project governance and similar discussions.  Maybe we should simply create a governance list and leave this one to be the free for all?

On Jul 2, 2011, at 9:16 PM, Ian Holsman wrote:

> 
> On Jul 2, 2011, at 7:38 AM, Ted Dunning wrote:
> 
>> On Fri, Jul 1, 2011 at 11:53 AM, Abhishek Mehta <ab...@tresata.com>wrote:
>> 
>>> open and honest conversations on this thread/group, irrespective of
>>> whether one represents a company or apache, should be encouraged.
>>> 
>> 
>> Paradoxically, I partially agree with Ian on this.  On the one hand, it is
>> important to hear about alternatives in the same eco-system (and alternative
>> approaches from other communities).  That is a bit different from Ian's
>> view.
> 
> While I don't mind that technical alternatives get discussed, I do get PO'd when
> the conversation goes into why product X is better than Y, or when someone makes claims that are incorrect because some 'customer' told them and stuff like that.
> 
> If we can keep it at architecture/approaches instead of why a certain product is better than go right ahead.
> 
>> 
>> Where I agree with him is that discussions should not be in the form of
>> simple breast beating.  Discussions should be driven by data and experience.
>> All data and experience are of equal value if they represent quality
>> thought.  The only place that company names should figure in this is as a
>> bookmark so you can tell which product/release has the characteristic under
>> consideration.
>> 
>> 
>>> ... And in that sense we all owe ASF and the hadoop community (and not any
>>> one company) an equal amount of gratitude, humility and respect.
>>> 
>> 
>> This doesn't get said nearly enough.
> 


Re: Hadoop Java Versions

Posted by Ian Holsman <ha...@holsman.net>.
On Jul 2, 2011, at 7:38 AM, Ted Dunning wrote:

> On Fri, Jul 1, 2011 at 11:53 AM, Abhishek Mehta <ab...@tresata.com>wrote:
> 
>> open and honest conversations on this thread/group, irrespective of
>> whether one represents a company or apache, should be encouraged.
>> 
> 
> Paradoxically, I partially agree with Ian on this.  On the one hand, it is
> important to hear about alternatives in the same eco-system (and alternative
> approaches from other communities).  That is a bit different from Ian's
> view.

While I don't mind that technical alternatives get discussed, I do get PO'd when
the conversation goes into why product X is better than Y, or when someone makes claims that are incorrect because some 'customer' told them and stuff like that.

If we can keep it at architecture/approaches instead of why a certain product is better than go right ahead.

> 
> Where I agree with him is that discussions should not be in the form of
> simple breast beating.  Discussions should be driven by data and experience.
> All data and experience are of equal value if they represent quality
> thought.  The only place that company names should figure in this is as a
> bookmark so you can tell which product/release has the characteristic under
> consideration.
> 
> 
>> ... And in that sense we all owe ASF and the hadoop community (and not any
>> one company) an equal amount of gratitude, humility and respect.
>> 
> 
> This doesn't get said nearly enough.


Re: Hadoop Java Versions

Posted by Ted Dunning <td...@maprtech.com>.
On Fri, Jul 1, 2011 at 11:53 AM, Abhishek Mehta <ab...@tresata.com>wrote:

>  open and honest conversations on this thread/group, irrespective of
> whether one represents a company or apache, should be encouraged.
>

Paradoxically, I partially agree with Ian on this.  On the one hand, it is
important to hear about alternatives in the same eco-system (and alternative
approaches from other communities).  That is a bit different from Ian's
view.

Where I agree with him is that discussions should not be in the form of
simple breast beating.  Discussions should be driven by data and experience.
 All data and experience are of equal value if they represent quality
thought.  The only place that company names should figure in this is as a
bookmark so you can tell which product/release has the characteristic under
consideration.


> ... And in that sense we all owe ASF and the hadoop community (and not any
> one company) an equal amount of gratitude, humility and respect.
>

This doesn't get said nearly enough.

Re: Hadoop Java Versions

Posted by Abhishek Mehta <ab...@tresata.com>.
i definitely agree with scott.  as a user of the hadoop open source stack for building our banking focused big data analytics applications, i speak on behalf of our clients and the emerging hadoop eco-system that open and honest conversations on this thread/group, irrespective of whether one represents a company or apache, should be encouraged.

as an instance, with the fact that cloudera, mapR and soon hortonworks are all going to be offering competing hadoop distros for enterprises, it is important for all of us (and prospective users) to understand what they are doing to address critical gaps on the platform, and how the hadoop ecosystem benefits from it.  

From our perspective, it doesn't matter if one is better than the other (which is not the point i saw ted or mc making), but that companies, startups, apache and everybody else:

1.  is thinking of the right issues
2.  willing to solve them (and ideally contributing the solutions back) and
3.  informing the exploding hadoop userbase of what not to do

I see it benefitting all of us, especially as Hadoop rapidly jumps the transom and becomes the platform of choice for data management in industries like banking, retail and healthcare...just as it has for social media and the web...

isn't that what we are launching our business plans around anyway...

And in that sense we all owe ASF and the hadoop community (and not any one company) an equal amount of gratitude, humility and respect.  


On Jul 1, 2011, at 1:22 PM, Scott Carey wrote:

> Although this thread is wandering a bit, I disagree strongly that it is
> inappropriate to discuss other vendor specific features (or competing
> compute platform features) on general@.  The topic has become the factors
> that influence hardware purchase choices, and one of those is how the
> system deals with disk failure.  Compare/contrast with other platforms is
> healthy for the Hadoop project! +1
> 
> On 6/30/11 9:47 PM, "Ian Holsman" <ha...@holsman.net> wrote:
> 
>> 
>> On Jul 1, 2011, at 2:08 PM, M. C. Srivas wrote:
>> 
>>> On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>> 
>>>> 
>>>> I'd advise you to look at "stock hadoop" again. This used to be true,
>>>> but
>>>> was fixed a long while back by HDFS-457 and several followup JIRAs.
>>>> 
>>>> If MapR does something fancier, I'm sure we'd be interested to hear
>>>> about
>>>> it
>>>> so we can compare the approaches.
>>>> 
>>>> -Todd
>>>> 
>>>> 
>>> MapR tracks disk responsiveness. In other words, a moving histogram of
>>> IO-completion times is maintained internally, and if a disk starts
>>> getting
>>> really slow, it is pre-emptively taken offline so it does not create
>>> long
>>> tails for running jobs (and the data on the disk is re-replicated using
>>> whatever re-replication policy is in place).  One of the benefits of
>>> managing the disks directly instead of through ext3 / xfs / or other ...
>>> 
>>> All these stats can be fed into Ganglia (or pushed out centrally via a
>>> text
>>> file that can be pulled out using NFS)  if historical info about disk
>>> behavior (and failures) needs to be preserved.
>>> 
>>> - Srivas.
>> 
>> While I am intrigued about how MapR performs internally, I don't think
>> this is the forum for it.
>> please keep MapR (and other vendor specific discussions) on their
>> respective support forums.
>> 
>> Thanks!
>> 
>> Ian.
>> 
> 


Re: Hadoop Java Versions

Posted by Scott Carey <sc...@richrelevance.com>.
Although this thread is wandering a bit, I disagree strongly that it is
inappropriate to discuss other vendor specific features (or competing
compute platform features) on general@.  The topic has become the factors
that influence hardware purchase choices, and one of those is how the
system deals with disk failure.  Compare/contrast with other platforms is
healthy for the Hadoop project!

On 6/30/11 9:47 PM, "Ian Holsman" <ha...@holsman.net> wrote:

>
>On Jul 1, 2011, at 2:08 PM, M. C. Srivas wrote:
>
>> On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon <to...@cloudera.com> wrote:
>> 
>>> 
>>> I'd advise you to look at "stock hadoop" again. This used to be true,
>>>but
>>> was fixed a long while back by HDFS-457 and several followup JIRAs.
>>> 
>>> If MapR does something fancier, I'm sure we'd be interested to hear
>>>about
>>> it
>>> so we can compare the approaches.
>>> 
>>> -Todd
>>> 
>>> 
>> MapR tracks disk responsiveness. In other words, a moving histogram of
>> IO-completion times is maintained internally, and if a disk starts
>>getting
>> really slow, it is pre-emptively taken offline so it does not create
>>long
>> tails for running jobs (and the data on the disk is re-replicated using
>> whatever re-replication policy is in place).  One of the benefits of
>> managing the disks directly instead of through ext3 / xfs / or other ...
>> 
>> All these stats can be fed into Ganglia (or pushed out centrally via a
>>text
>> file that can be pulled out using NFS)  if historical info about disk
>> behavior (and failures) needs to be preserved.
>> 
>> - Srivas.
>
>While I am intrigued about how MapR performs internally, I don't think
>this is the forum for it.
>please keep MapR (and other vendor specific discussions) on their
>respective support forums.
>
>Thanks!
>
>Ian.
>


Re: Hadoop Java Versions

Posted by Ian Holsman <ha...@holsman.net>.
On Jul 1, 2011, at 2:08 PM, M. C. Srivas wrote:

> On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon <to...@cloudera.com> wrote:
> 
>> 
>> I'd advise you to look at "stock hadoop" again. This used to be true, but
>> was fixed a long while back by HDFS-457 and several followup JIRAs.
>> 
>> If MapR does something fancier, I'm sure we'd be interested to hear about
>> it
>> so we can compare the approaches.
>> 
>> -Todd
>> 
>> 
> MapR tracks disk responsiveness. In other words, a moving histogram of
> IO-completion times is maintained internally, and if a disk starts getting
> really slow, it is pre-emptively taken offline so it does not create long
> tails for running jobs (and the data on the disk is re-replicated using
> whatever re-replication policy is in place).  One of the benefits of
> managing the disks directly instead of through ext3 / xfs / or other ...
> 
> All these stats can be fed into Ganglia (or pushed out centrally via a text
> file that can be pulled out using NFS)  if historical info about disk
> behavior (and failures) needs to be preserved.
> 
> - Srivas.

While I am intrigued about how MapR performs internally, I don't think this is the forum for it.
please keep MapR (and other vendor specific discussions) on their respective support forums.

Thanks!

Ian.


Re: Hadoop Java Versions

Posted by "M. C. Srivas" <mc...@gmail.com>.
On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon <to...@cloudera.com> wrote:

>
> I'd advise you to look at "stock hadoop" again. This used to be true, but
> was fixed a long while back by HDFS-457 and several followup JIRAs.
>
> If MapR does something fancier, I'm sure we'd be interested to hear about
> it
> so we can compare the approaches.
>
> -Todd
>
>
MapR tracks disk responsiveness. In other words, a moving histogram of
IO-completion times is maintained internally, and if a disk starts getting
really slow, it is pre-emptively taken offline so it does not create long
tails for running jobs (and the data on the disk is re-replicated using
whatever re-replication policy is in place).  One of the benefits of
managing the disks directly instead of through ext3 / xfs / or other ...

All these stats can be fed into Ganglia (or pushed out centrally via a text
file that can be pulled out using NFS)  if historical info about disk
behavior (and failures) needs to be preserved.

- Srivas.

Re: Hadoop Java Versions

Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Jun 30, 2011 at 5:16 PM, Ted Dunning <td...@maprtech.com> wrote:

> You have to consider the long-term reliability as well.
>
> Losing an entire set of 10 or 12 disks at once makes the overall
> reliability
> of a large cluster very suspect.  This is because it becomes entirely too
> likely that two additional drives will fail before the data on the off-line
> node can be replicated.  For 100 nodes, that can decrease the average time
> to data loss down to less than a year.  This can only be mitigated in stock
> hadoop by keeping the number of drives relatively low.  MapR avoids this by
> not failing nodes for trivial problems.
>

I'd advise you to look at "stock hadoop" again. This used to be true, but
was fixed a long while back by HDFS-457 and several followup JIRAs.

If MapR does something fancier, I'm sure we'd be interested to hear about it
so we can compare the approaches.

-Todd


>
> On Thu, Jun 30, 2011 at 4:18 PM, Aaron Eng <ae...@maprtech.com> wrote:
>
> > >Keeping the amount of disks per node low and the amount of nodes high
> > should keep the impact of dead nodes in control.
> >
> > It keeps the impact of dead nodes in control but I don't think thats
> > long-term cost efficient.  As prices of 10GbE go down, the "keep the node
> > small" arguement seems less fitting.  And on another note, most servers
> > manufactured in the last 10 years have dual 1GbE network interfaces.  If
> > one
> > were to go by these calcs:
> >
> > >150 nodes with four 2TB disks each, with HDFS 60% full, it takes around
> > ~32
> > minutes to recover
> >
> > It seems like that assumes a single 1GbE interface, why  not leverage the
> > second?
> >
> > On Thu, Jun 30, 2011 at 2:31 PM, Evert Lammerts <Evert.Lammerts@sara.nl
> > >wrote:
> >
> > > > You can get 12-24 TB in a server today, which means the loss of a
> > server
> > > > generates a lot of traffic -which argues for 10 Gbe.
> > > >
> > > > But
> > > >   -big increase in switch cost, especially if you (CoI warning) go
> with
> > > > Cisco
> > > >   -there have been problems with things like BIOS PXE and lights out
> > > > management on 10 Gbe -probably due to the NICs being things the BIOS
> > > > wasn't expecting and off the mainboard. This should improve.
> > > >   -I don't know how well linux works with ether that fast (field
> > reports
> > > > useful)
> > > >   -the big threat is still ToR switch failure, as that will trigger a
> > > > re-replication of every block in the rack.
> > >
> > > Keeping the amount of disks per node low and the amount of nodes high
> > > should keep the impact of dead nodes in control. A ToR switch failing
> is
> > > different - missing 30 nodes (~120TB) at once cannot be fixed by adding
> > more
> > > nodes; that actually increases ToR switch failure. Although such
> failure
> > is
> > > quite rare to begin with, I guess. The back-of-the-envelope-calculation
> I
> > > made suggests that ~150 (1U) nodes should be fine with 1Gb ethernet.
> > (e.g.,
> > > when 6 nodes fail in a cluster with 150 nodes with four 2TB disks each,
> > with
> > > HDFS 60% full, it takes around ~32 minutes to recover. 2 nodes failing
> > > should take around 640 seconds. Also see the attached spreadsheet.)
> This
> > > doesn't take ToR switch failure in account though. On the other hand -
> > 150
> > > nodes is only ~5 racks - in such a scenario you might rather want to
> shut
> > > the system down completely rather than letting it replicate 20% of all
> > data.
> > >
> > > Cheers,
> > > Evert
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Hadoop Java Versions

Posted by Ted Dunning <td...@maprtech.com>.
You have to consider the long-term reliability as well.

Losing an entire set of 10 or 12 disks at once makes the overall reliability
of a large cluster very suspect.  This is because it becomes entirely too
likely that two additional drives will fail before the data on the off-line
node can be replicated.  For 100 nodes, that can decrease the average time
to data loss down to less than a year.  This can only be mitigated in stock
hadoop by keeping the number of drives relatively low.  MapR avoids this by
not failing nodes for trivial problems.

On Thu, Jun 30, 2011 at 4:18 PM, Aaron Eng <ae...@maprtech.com> wrote:

> >Keeping the amount of disks per node low and the amount of nodes high
> should keep the impact of dead nodes in control.
>
> It keeps the impact of dead nodes in control but I don't think thats
> long-term cost efficient.  As prices of 10GbE go down, the "keep the node
> small" arguement seems less fitting.  And on another note, most servers
> manufactured in the last 10 years have dual 1GbE network interfaces.  If
> one
> were to go by these calcs:
>
> >150 nodes with four 2TB disks each, with HDFS 60% full, it takes around
> ~32
> minutes to recover
>
> It seems like that assumes a single 1GbE interface, why  not leverage the
> second?
>
> On Thu, Jun 30, 2011 at 2:31 PM, Evert Lammerts <Evert.Lammerts@sara.nl
> >wrote:
>
> > > You can get 12-24 TB in a server today, which means the loss of a
> server
> > > generates a lot of traffic -which argues for 10 Gbe.
> > >
> > > But
> > >   -big increase in switch cost, especially if you (CoI warning) go with
> > > Cisco
> > >   -there have been problems with things like BIOS PXE and lights out
> > > management on 10 Gbe -probably due to the NICs being things the BIOS
> > > wasn't expecting and off the mainboard. This should improve.
> > >   -I don't know how well linux works with ether that fast (field
> reports
> > > useful)
> > >   -the big threat is still ToR switch failure, as that will trigger a
> > > re-replication of every block in the rack.
> >
> > Keeping the amount of disks per node low and the amount of nodes high
> > should keep the impact of dead nodes in control. A ToR switch failing is
> > different - missing 30 nodes (~120TB) at once cannot be fixed by adding
> more
> > nodes; that actually increases ToR switch failure. Although such failure
> is
> > quite rare to begin with, I guess. The back-of-the-envelope-calculation I
> > made suggests that ~150 (1U) nodes should be fine with 1Gb ethernet.
> (e.g.,
> > when 6 nodes fail in a cluster with 150 nodes with four 2TB disks each,
> with
> > HDFS 60% full, it takes around ~32 minutes to recover. 2 nodes failing
> > should take around 640 seconds. Also see the attached spreadsheet.) This
> > doesn't take ToR switch failure in account though. On the other hand -
> 150
> > nodes is only ~5 racks - in such a scenario you might rather want to shut
> > the system down completely rather than letting it replicate 20% of all
> data.
> >
> > Cheers,
> > Evert
>