You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Vincent Boucher <vi...@gmail.com> on 2011/10/11 10:39:46 UTC
blocks with a huge size?
Hello again,
* Our case:
Most of the files we are dealing with are 10GB wide. Our hdfs configuration
would be the following: data is stored on mass storage servers (10x50TB) each
with RAID6; no replica for data.
With a 64MB hdfs block size, it is extremely likely that all of our 10GB files
will be spread over all the mass storage servers. Consequently, having one of
these servers down/dead will corrupt the full filesystem (all the 10GB
files). Not great.
Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
spread: the file contents will be stored on a single server. Having one
server down/dead will corrupt only 10% of the files in the filesystem (since
there are 10 servers). That's much easier to regenerate/re-download from
other Tiers than doing it for the full filesystem, as in the case of the 64MB
blocks.
* Questions:
Is hdfs suitable with huge block size (12.5GB)?
Do you have experience with hdfs with such block size?
Cheers,
Vincent
Re: blocks with a huge size?
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Vincent,
Maybe you'd be better off using something like MogileFS for this application?
-Todd
On Tue, Oct 11, 2011 at 1:39 AM, Vincent Boucher <vi...@gmail.com> wrote:
> Hello again,
>
> * Our case:
>
> Most of the files we are dealing with are 10GB wide. Our hdfs configuration
> would be the following: data is stored on mass storage servers (10x50TB) each
> with RAID6; no replica for data.
>
> With a 64MB hdfs block size, it is extremely likely that all of our 10GB files
> will be spread over all the mass storage servers. Consequently, having one of
> these servers down/dead will corrupt the full filesystem (all the 10GB
> files). Not great.
>
> Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
> spread: the file contents will be stored on a single server. Having one
> server down/dead will corrupt only 10% of the files in the filesystem (since
> there are 10 servers). That's much easier to regenerate/re-download from
> other Tiers than doing it for the full filesystem, as in the case of the 64MB
> blocks.
>
>
> * Questions:
>
> Is hdfs suitable with huge block size (12.5GB)?
>
> Do you have experience with hdfs with such block size?
>
>
> Cheers,
>
> Vincent
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: blocks with a huge size?
Posted by Vincent Boucher <vi...@gmail.com>.
Le Tuesday 11 October 2011, Sudharsan Sampath a écrit :
> Hello,
>
> I donot understand "having one server down will corrupt the full
> filesystem". HDFS allows for high availability based on the replication
> factor that you set.
>
> Do you have a constraint on setting the replication factor more than 1 ?
Yes, we can't afford a replication factor of 2 (data size = 500TB), we
mitigate the risk of datacorruption with raid 6 arrays (or raidz2). We are
looking for a strategy in Hadoop to reduce the risk again by enhancing the
size of the blocks. Although the data can either be downloaded again from
another remote storage site or recalculated, it would need a week to recover
the full filesystem with a 10Gbps link, so we'd prefer recovering 10% instead
of 100% ;-)
> Thanks
> Sudharsan S
>
> On Tue, Oct 11, 2011 at 2:09 PM, Vincent Boucher
<vi...@gmail.com>wrote:
> > Hello again,
> >
> > * Our case:
> >
> > Most of the files we are dealing with are 10GB wide. Our hdfs
> > configuration would be the following: data is stored on mass storage
> > servers (10x50TB) each
> > with RAID6; no replica for data.
> >
> > With a 64MB hdfs block size, it is extremely likely that all of our 10GB
> > files
> > will be spread over all the mass storage servers. Consequently, having
> > one of
> > these servers down/dead will corrupt the full filesystem (all the 10GB
> > files). Not great.
> >
> > Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
> > spread: the file contents will be stored on a single server. Having one
> > server down/dead will corrupt only 10% of the files in the filesystem
> > (since
> > there are 10 servers). That's much easier to regenerate/re-download from
> > other Tiers than doing it for the full filesystem, as in the case of the
> > 64MB
> > blocks.
> >
> >
> > * Questions:
> >
> > Is hdfs suitable with huge block size (12.5GB)?
> >
> > Do you have experience with hdfs with such block size?
> >
> >
> > Cheers,
> >
> > Vincent
Re: blocks with a huge size?
Posted by Sudharsan Sampath <su...@gmail.com>.
Hello,
I donot understand "having one server down will corrupt the full
filesystem". HDFS allows for high availability based on the replication
factor that you set.
Do you have a constraint on setting the replication factor more than 1 ?
Thanks
Sudharsan S
On Tue, Oct 11, 2011 at 2:09 PM, Vincent Boucher <vi...@gmail.com>wrote:
> Hello again,
>
> * Our case:
>
> Most of the files we are dealing with are 10GB wide. Our hdfs configuration
> would be the following: data is stored on mass storage servers (10x50TB)
> each
> with RAID6; no replica for data.
>
> With a 64MB hdfs block size, it is extremely likely that all of our 10GB
> files
> will be spread over all the mass storage servers. Consequently, having one
> of
> these servers down/dead will corrupt the full filesystem (all the 10GB
> files). Not great.
>
> Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
> spread: the file contents will be stored on a single server. Having one
> server down/dead will corrupt only 10% of the files in the filesystem
> (since
> there are 10 servers). That's much easier to regenerate/re-download from
> other Tiers than doing it for the full filesystem, as in the case of the
> 64MB
> blocks.
>
>
> * Questions:
>
> Is hdfs suitable with huge block size (12.5GB)?
>
> Do you have experience with hdfs with such block size?
>
>
> Cheers,
>
> Vincent
>
Re: blocks with a huge size?
Posted by Will Maier <wc...@hep.wisc.edu>.
Hi Vincent-
On Tue, Oct 11, 2011 at 07:12:55PM +0200, Vincent Boucher wrote:
> Indeed, it's a similar setup, at a smaller scale: Tier2 for CMS at UCLouvain,
> Belgium, with data&sim from the whole collaboration; stageout files from the
> Grid for our local users ...
It's always nice to see other CMS sites get involved with HDFS!
[...]
> For us, it is difficult to set a 2x replication while maintaining the current
> amount of data we serve to the Collaboration.
>
> An alternative would be to switch the mass storage servers with currently one
> large ZFS partition each to servers with as many independent partitions as the
> # of drives they host (typical # of drives per server: 70); and set at the
> Hadoop level a 2x replication. The volume freed by killing the RAID is not
> enough to compensate the replication but that would be a first step.
In fact, we run with exactly this configuration on our larger servers: each disk
is its own RAID0 with a single partition (and 2x replication overall). This
JBOD style is sensitive to disk failure, though the upcoming version is less so.
At our site, we work around the issue by dynamically reconfiguring the
dfs.data.dir parameter in hdfs-site.xml:
http://hg.hep.wisc.edu/cmsops/hdfs/file/tip/bin/dfs-datadir.sh
Feel free to contact me offlist if you have questions more specific to running
HDFS in CMS.
Good luck!
--
Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/
Re: blocks with a huge size?
Posted by Vincent Boucher <vi...@gmail.com>.
Hi Will-
Le Tuesday 11 October 2011, Will Maier a écrit :
> As I described earlier, our ~1PB cluster is very similar: some large
> servers, some small ones. We store data recorded at and simulated for the
> CMS detector at the LHC, as well as the products of our users' physics
> analysis. Like you, much of our data can be redownloaded to our Tier2 from
> the nearest Tier1 (Fermilab, USA).
Indeed, it's a similar setup, at a smaller scale: Tier2 for CMS at UCLouvain,
Belgium, with data&sim from the whole collaboration; stageout files from the
Grid for our local users ...
> And still, we've gone with a fairly standard configuration, with 2x
> replication across the board. I strongly suggest trying to design your
> cluster to exploit the good parts of HDFS instead of making it look like
> whatever your previous system was (ours was dCache, a distributed disk
> cache developed in and for the high energy physics community). HDFS doesn't
> map replicas to servers and works best with a replication factor >=2. These
> were new constraints for us when we migrated our storage cluster, but the
> benefits of storing data on commodity servers (our worker nodes) with
> inbuilt replication/resiliency and great throughput more than compensated.
We were working with a kind of homemade FUSE solution built on top of NFS
(with a Metadata server hosting symbolic links to the different mass storage
servers). The data was sprinkled over the mass storage servers at the file
level, not block level.
It achieved great performance and worked very well as far as everything
was ... working well. Eg: when a NFS server fails, the clients have a strange
behaviour (whole nfs hanging). No clean admin tools.
For us, it is difficult to set a 2x replication while maintaining the current
amount of data we serve to the Collaboration.
An alternative would be to switch the mass storage servers with currently one
large ZFS partition each to servers with as many independent partitions as
the # of drives they host (typical # of drives per server: 70); and set at
the Hadoop level a 2x replication. The volume freed by killing the RAID is
not enough to compensate the replication but that would be a first step.
Vincent
Re: blocks with a huge size?
Posted by Vincent Boucher <vi...@gmail.com>.
Hi Matt-
Le Tuesday 11 October 2011, GOEKE, MATTHEW (AG/1000) a écrit :
> Vincent,
>
> Just to clarify, are you using hdfs/Hadoop primarily as a storage layer
> with little to no MR? I know that is the case for Will's team and I was
> kind of assuming that as another tier2 you would probably be doing the
> same. The reason why I ask is the design decisions you have made so far
> have large implications on the throughput of your MR jobs.
Yes, we are using Hadoop for its storage capacities only, not for the
MapReduce tasks. The job management is performed by Condor (and we're
thinking about moving to Slurm). Currently, the job manager has no idea about
data locality. Note that datasets are chunked in files of ~10GB and a
analysis process -typically- seeks one of these file.
Vincent
>
> Matt
>
> -----Original Message-----
> From: Will Maier [mailto:wcmaier@hep.wisc.edu]
> Sent: Tuesday, October 11, 2011 5:43 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: blocks with a huge size?
>
> Hi Vincent-
>
> On Tue, Oct 11, 2011 at 10:39:46AM +0200, Vincent Boucher wrote:
> > Most of the files we are dealing with are 10GB wide. Our hdfs
> > configuration would be the following: data is stored on mass storage
> > servers (10x50TB) each with RAID6; no replica for data.
> >
> > With a 64MB hdfs block size, it is extremely likely that all of our 10GB
> > files will be spread over all the mass storage servers. Consequently,
> > having one of these servers down/dead will corrupt the full filesystem
> > (all the 10GB files). Not great.
> >
> > Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
> > spread: the file contents will be stored on a single server. Having one
> > server down/dead will corrupt only 10% of the files in the filesystem
> > (since there are 10 servers). That's much easier to
> > regenerate/re-download from other Tiers than doing it for the full
> > filesystem, as in the case of the 64MB blocks.
>
> As I described earlier, our ~1PB cluster is very similar: some large
> servers, some small ones. We store data recorded at and simulated for the
> CMS detector at the LHC, as well as the products of our users' physics
> analysis. Like you, much of our data can be redownloaded to our Tier2 from
> the nearest Tier1 (Fermilab, USA).
>
> And still, we've gone with a fairly standard configuration, with 2x
> replication across the board. I strongly suggest trying to design your
> cluster to exploit the good parts of HDFS instead of making it look like
> whatever your previous system was (ours was dCache, a distributed disk
> cache developed in and for the high energy physics community). HDFS doesn't
> map replicas to servers and works best with a replication factor >=2. These
> were new constraints for us when we migrated our storage cluster, but the
> benefits of storing data on commodity servers (our worker nodes) with
> inbuilt replication/resiliency and great throughput more than compensated.
RE: blocks with a huge size?
Posted by "GOEKE, MATTHEW (AG/1000)" <ma...@monsanto.com>.
Vincent,
Just to clarify, are you using hdfs/Hadoop primarily as a storage layer with little to no MR? I know that is the case for Will's team and I was kind of assuming that as another tier2 you would probably be doing the same. The reason why I ask is the design decisions you have made so far have large implications on the throughput of your MR jobs.
Matt
-----Original Message-----
From: Will Maier [mailto:wcmaier@hep.wisc.edu]
Sent: Tuesday, October 11, 2011 5:43 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: blocks with a huge size?
Hi Vincent-
On Tue, Oct 11, 2011 at 10:39:46AM +0200, Vincent Boucher wrote:
> Most of the files we are dealing with are 10GB wide. Our hdfs configuration
> would be the following: data is stored on mass storage servers (10x50TB) each
> with RAID6; no replica for data.
>
> With a 64MB hdfs block size, it is extremely likely that all of our 10GB files
> will be spread over all the mass storage servers. Consequently, having one of
> these servers down/dead will corrupt the full filesystem (all the 10GB files).
> Not great.
>
> Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
> spread: the file contents will be stored on a single server. Having one server
> down/dead will corrupt only 10% of the files in the filesystem (since there
> are 10 servers). That's much easier to regenerate/re-download from other Tiers
> than doing it for the full filesystem, as in the case of the 64MB blocks.
As I described earlier, our ~1PB cluster is very similar: some large servers,
some small ones. We store data recorded at and simulated for the CMS detector at
the LHC, as well as the products of our users' physics analysis. Like you, much
of our data can be redownloaded to our Tier2 from the nearest Tier1 (Fermilab,
USA).
And still, we've gone with a fairly standard configuration, with 2x replication
across the board. I strongly suggest trying to design your cluster to exploit
the good parts of HDFS instead of making it look like whatever your previous
system was (ours was dCache, a distributed disk cache developed in and for the
high energy physics community). HDFS doesn't map replicas to servers and works
best with a replication factor >=2. These were new constraints for us when we
migrated our storage cluster, but the benefits of storing data on commodity
servers (our worker nodes) with inbuilt replication/resiliency and great
throughput more than compensated.
--
Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.
The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.
Re: blocks with a huge size?
Posted by Will Maier <wc...@hep.wisc.edu>.
Hi Vincent-
On Tue, Oct 11, 2011 at 10:39:46AM +0200, Vincent Boucher wrote:
> Most of the files we are dealing with are 10GB wide. Our hdfs configuration
> would be the following: data is stored on mass storage servers (10x50TB) each
> with RAID6; no replica for data.
>
> With a 64MB hdfs block size, it is extremely likely that all of our 10GB files
> will be spread over all the mass storage servers. Consequently, having one of
> these servers down/dead will corrupt the full filesystem (all the 10GB files).
> Not great.
>
> Opting for bigger blocks (blocks of 12.5GB [= 200x64MB]) will reduce the
> spread: the file contents will be stored on a single server. Having one server
> down/dead will corrupt only 10% of the files in the filesystem (since there
> are 10 servers). That's much easier to regenerate/re-download from other Tiers
> than doing it for the full filesystem, as in the case of the 64MB blocks.
As I described earlier, our ~1PB cluster is very similar: some large servers,
some small ones. We store data recorded at and simulated for the CMS detector at
the LHC, as well as the products of our users' physics analysis. Like you, much
of our data can be redownloaded to our Tier2 from the nearest Tier1 (Fermilab,
USA).
And still, we've gone with a fairly standard configuration, with 2x replication
across the board. I strongly suggest trying to design your cluster to exploit
the good parts of HDFS instead of making it look like whatever your previous
system was (ours was dCache, a distributed disk cache developed in and for the
high energy physics community). HDFS doesn't map replicas to servers and works
best with a replication factor >=2. These were new constraints for us when we
migrated our storage cluster, but the benefits of storing data on commodity
servers (our worker nodes) with inbuilt replication/resiliency and great
throughput more than compensated.
--
Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/