You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Touretsky, Gregory" <gr...@intel.com> on 2009/09/16 09:09:13 UTC
Stretched HDFS cluster
Hi,
Does anyone have an experience running HDFS cluster stretched over high-latency WAN connections?
Any specific concerns/options/recommendations?
I'm trying to setup the HDFS cluster with the nodes located in the US, Israel and India - considering it as a potential solution for cross-site data sharing...
Regards,
Gregory Touretsky
Intel IT - Strategic Solutions and Architecture
Systems Analyst
gregory.touretsky AT intel.com
(+) 972-4-865-6377, Fax: 04-865-5999
iNET: 465-6377, M/S: IDC10-2.3
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Re: Stretched HDFS cluster
Posted by Steve Loughran <st...@apache.org>.
Edward Capriolo wrote:
>>
> On a somewhat related topic I was showing a co-worker a Hadoop setup
> and he asked stated, "What if we got a bunch of laptops on the
> internet like the playstation 'Folding @ Home'" of course these are
> widely different distributed models.
>
> I have been thinking about this. Assume:
> Throw the data security out the window, and assume everyone is playing fair.
> Assume we have systems with a semi-dedicated IP, like my cable
> internet. with no inbound/outbound restrictions.
> Assume every computer is its own RACK
> LAN is very low latency Assume that latency is like 40 ms
> Assume we tune up replication to say 10 or higher to deal with drop on/drop offs
>
> Could we run a 100 Node cluster? If no what is stopping it from working?
>
> My next question. For fun, does anyone want to try? We could setup
> IPTABLES/firewall allowing hadoop traffic from IP's in the experiment.
> I have two nodes in Chicago, US ISP to donate. Even if we get 10 nodes
> that would be interesting as a benchmark.
you could see about getting planet-lab time for something like this,
though the way they allocate partially virtualized slices, you can't be
100% sure that you get the ports you ask for. This will complicate binding.
Re: Stretched HDFS cluster
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Sep 16, 2009 at 7:27 AM, Steve Loughran <st...@apache.org> wrote:
> Touretsky, Gregory wrote:
>>
>> Hi,
>>
>> Does anyone have an experience running HDFS cluster stretched over
>> high-latency WAN connections?
>> Any specific concerns/options/recommendations?
>> I'm trying to setup the HDFS cluster with the nodes located in the US,
>> Israel and India - considering it as a potential solution for cross-site
>> data sharing...
>>
>
> I would back up todd here and say "don't do it -yet". I think there are some
> minor placeholders in the rack hierarchy to have an explicit notion of
> different sites, but nobody has done the work yet. Cross datacentre data
> balancing and work scheduling is complex, and all the code in Hadoop,
> zookeeper, etc, is built on the assumption that latency is low, all machines
> clocks are going forward at roughly the same rate, the network is fairly
> reliable, routers are unlikely to corrupt data, etc.
>
> Now, if you do want to do >1 site, it would be a profound and useful
> development -I'd expect the MR scheduler, or even the Pig/Hive code
> generators , to take datacentre locality into account, doing as much work
> per site as possible. The problem of block distribution changes too, as you
> would want 1 copy of each block in the other datacentre. Even then, I'd
> start with sites in a single city, on a MAE or other link where bandwidth
> matters less. Note that (as discussed below) on the MAN scale things can
> start to go wrong in ways that are unlikely in a datacentre, and its those
> failures that will burn you
>
> worth reading
> http://status.aws.amazon.com/s3-20080720.html
> http://www.allthingsdistributed.com/2008/12/eventually_consistent.html
>
> -Steve
>
On a somewhat related topic I was showing a co-worker a Hadoop setup
and he asked stated, "What if we got a bunch of laptops on the
internet like the playstation 'Folding @ Home'" of course these are
widely different distributed models.
I have been thinking about this. Assume:
Throw the data security out the window, and assume everyone is playing fair.
Assume we have systems with a semi-dedicated IP, like my cable
internet. with no inbound/outbound restrictions.
Assume every computer is its own RACK
LAN is very low latency Assume that latency is like 40 ms
Assume we tune up replication to say 10 or higher to deal with drop on/drop offs
Could we run a 100 Node cluster? If no what is stopping it from working?
My next question. For fun, does anyone want to try? We could setup
IPTABLES/firewall allowing hadoop traffic from IP's in the experiment.
I have two nodes in Chicago, US ISP to donate. Even if we get 10 nodes
that would be interesting as a benchmark.
Re: Stretched HDFS cluster
Posted by Steve Loughran <st...@apache.org>.
Touretsky, Gregory wrote:
> Hi,
>
> Does anyone have an experience running HDFS cluster stretched over high-latency WAN connections?
> Any specific concerns/options/recommendations?
> I'm trying to setup the HDFS cluster with the nodes located in the US, Israel and India - considering it as a potential solution for cross-site data sharing...
>
I would back up todd here and say "don't do it -yet". I think there are
some minor placeholders in the rack hierarchy to have an explicit notion
of different sites, but nobody has done the work yet. Cross datacentre
data balancing and work scheduling is complex, and all the code in
Hadoop, zookeeper, etc, is built on the assumption that latency is low,
all machines clocks are going forward at roughly the same rate, the
network is fairly reliable, routers are unlikely to corrupt data, etc.
Now, if you do want to do >1 site, it would be a profound and useful
development -I'd expect the MR scheduler, or even the Pig/Hive code
generators , to take datacentre locality into account, doing as much
work per site as possible. The problem of block distribution changes
too, as you would want 1 copy of each block in the other datacentre.
Even then, I'd start with sites in a single city, on a MAE or other link
where bandwidth matters less. Note that (as discussed below) on the MAN
scale things can start to go wrong in ways that are unlikely in a
datacentre, and its those failures that will burn you
worth reading
http://status.aws.amazon.com/s3-20080720.html
http://www.allthingsdistributed.com/2008/12/eventually_consistent.html
-Steve
Re: Stretched HDFS cluster
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Gregory,
This is way outside the design parameters of HDFS. It may work, but you are
very likely to run into issues, and I don't think anyone would recommend
this as a solution. More reasonable would be a HDFS cluster spanning two
datacenters within the same metro area (1-2ms latency), but even that isn't
at all common.
For cross-site redundancy of HDFS, the usual solution is using distcp to
make periodic backups between separate clusters.
-Todd
On Wed, Sep 16, 2009 at 12:09 AM, Touretsky, Gregory <
gregory.touretsky@intel.com> wrote:
> Hi,
>
> Does anyone have an experience running HDFS cluster stretched over
> high-latency WAN connections?
> Any specific concerns/options/recommendations?
> I'm trying to setup the HDFS cluster with the nodes located in the US,
> Israel and India - considering it as a potential solution for cross-site
> data sharing...
>
> Regards,
> Gregory Touretsky
> Intel IT - Strategic Solutions and Architecture
> Systems Analyst
> gregory.touretsky AT intel.com
> (+) 972-4-865-6377, Fax: 04-865-5999
> iNET: 465-6377, M/S: IDC10-2.3
>
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>