You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by "Taylor, Ronald C" <ro...@pnl.gov> on 2010/12/28 23:04:48 UTC
What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Kevin Fox <Ke...@pnl.gov>.
As I understand it, and please correct me if I'm wrong, a map/reduce job
has an instance of a FileSystem object on either side. One that the data
is read out of on the map side, and one the data is fed into on the
reduce side.
Can't you run the map reduce job on the storage cluster that stores the
archival data, feeding the map side with
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/RawLocalFileSystem.html
from the mounted, posix parallel file system,
and feed it into the hadoop cluster on the reduce side. That would mean,
the network in the middle would only see the reduced data set cross the
wire and you could parallelize the data reduction as close to the
archive as possible.
Thanks,
Kevin
On Tue, 2010-12-28 at 14:04 -0800, Taylor, Ronald C wrote:
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Patrick Angeles <pa...@cloudera.com>.
To add to what Ted says,
"hadoop fs -copyFromLocal" assumes that the file is present locally on the
datanode. That means the file would have to have been transfered to the
node, copied to local disk, and only after that is it written to HDFS, so
there's an extra trip to disk that you could have avoided.
You might have avoided that using a direct pull from the MapReduce task that
writes directly to HDFS. But, as Ted mentions, that is not as efficient as a
source-based push and is also more complex.
On Tue, Dec 28, 2010 at 6:07 PM, Ted Dunning <td...@maprtech.com> wrote:
> if the data is coming off of a single machine then simply running multiple
> threads on that machine spraying the data into the cluster is likely to be
> faster than a map-reduce program. The reason is that you can run the
> spraying process continuously and can tune it to carefully saturate your
> outbound link toward the cluster. With a map-reduce program it will be very
> easy to flatten the link.
>
> Another issue is that it is easy to push data to the cluster from a local
> disk rather than to pull it from nodes in the cluster because most network
> file protocols aren't as efficient as you might like.
>
>
> On Tue, Dec 28, 2010 at 2:47 PM, Taylor, Ronald C <ro...@pnl.gov>wrote:
>
>> 2) some way of parallelizing the reads
>>
>> So - I will check into network hardware, in regard to (1). But for (2), is
>> the MapReduce method that I was think of, a way that uses "hadoop fs
>> -copyFromLocal" in each Mapper, a good way to go at the destination end? I
>> believe that you were saying that it is indeed OK, but I want to
>> double-check, since this will be a critical piece of our work flow.
>>
>
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Ted Dunning <td...@maprtech.com>.
if the data is coming off of a single machine then simply running multiple
threads on that machine spraying the data into the cluster is likely to be
faster than a map-reduce program. The reason is that you can run the
spraying process continuously and can tune it to carefully saturate your
outbound link toward the cluster. With a map-reduce program it will be very
easy to flatten the link.
Another issue is that it is easy to push data to the cluster from a local
disk rather than to pull it from nodes in the cluster because most network
file protocols aren't as efficient as you might like.
On Tue, Dec 28, 2010 at 2:47 PM, Taylor, Ronald C <ro...@pnl.gov>wrote:
> 2) some way of parallelizing the reads
>
> So - I will check into network hardware, in regard to (1). But for (2), is
> the MapReduce method that I was think of, a way that uses "hadoop fs
> -copyFromLocal" in each Mapper, a good way to go at the destination end? I
> believe that you were saying that it is indeed OK, but I want to
> double-check, since this will be a critical piece of our work flow.
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Ted Dunning <td...@maprtech.com>.
if the data is coming off of a single machine then simply running multiple
threads on that machine spraying the data into the cluster is likely to be
faster than a map-reduce program. The reason is that you can run the
spraying process continuously and can tune it to carefully saturate your
outbound link toward the cluster. With a map-reduce program it will be very
easy to flatten the link.
Another issue is that it is easy to push data to the cluster from a local
disk rather than to pull it from nodes in the cluster because most network
file protocols aren't as efficient as you might like.
On Tue, Dec 28, 2010 at 2:47 PM, Taylor, Ronald C <ro...@pnl.gov>wrote:
> 2) some way of parallelizing the reads
>
> So - I will check into network hardware, in regard to (1). But for (2), is
> the MapReduce method that I was think of, a way that uses "hadoop fs
> -copyFromLocal" in each Mapper, a good way to go at the destination end? I
> believe that you were saying that it is indeed OK, but I want to
> double-check, since this will be a critical piece of our work flow.
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Taylor, Ronald C" <ro...@pnl.gov>.
Patrick,
Thanks for the info (and quick reply). I want to make sure I understand: Presuming that the data files are coming off a set of disk drives attached to a single Linux file server, you say I need two things to optimize the transfer:
1) a fat network pipe
2) some way of parallelizing the reads
So - I will check into network hardware, in regard to (1). But for (2), is the MapReduce method that I was think of, a way that uses "hadoop fs -copyFromLocal" in each Mapper, a good way to go at the destination end? I believe that you were saying that it is indeed OK, but I want to double-check, since this will be a critical piece of our work flow.
Ron
________________________________
From: patrickangeles@gmail.com [mailto:patrickangeles@gmail.com] On Behalf Of Patrick Angeles
Sent: Tuesday, December 28, 2010 2:27 PM
To: general@hadoop.apache.org
Cc: user@hbase.apache.org; Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Ron,
While MapReduce can help to parallelize the load effort, your likely bottleneck is the source system (where the files come from). If the files are coming from a single server, then parallelizing the load won't gain you much past a certain point. You have to figure in how fast you can read the file(s) off disk(s) and push the bits through your network and finally onto HDFS.
The best scenario is if you can parallelize the reads and have a fat network pipe (10GbE or more) going into your Hadoop cluster.
Regards,
- Patrick
On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C <ro...@pnl.gov>> wrote:
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov<ma...@pnl.gov>
RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Posted by "Hiller, Dean (Contractor)" <de...@broadridge.com>.
Thanks of the info, missed that at the bottom of that page.
Dean
-----Original Message-----
From: Fox, Kevin M [mailto:kevin.fox@pnl.gov]
Sent: Wednesday, December 29, 2010 2:21 PM
To: Hiller, Dean (Contractor); general@hadoop.apache.org; Patrick
Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
http://wiki.apache.org/hadoop/MountableHDFS
Under Known Issues:
2. Writes are approximately 33% slower than the DFSClient. TBD how to
optimize this. see: HADOOP-3805 - try using -obig_writes if on a
>2.6.26 kernel, should perform much better since bigger writes implies
less context switching.
3. Reads are ~20-30% slower even with the read buffering.
Sounds like just pushing it in would be better.
Thanks,
Kevin
-----Original Message-----
From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]
Sent: Wednesday, December 29, 2010 1:16 PM
To: general@hadoop.apache.org; Fox, Kevin M; Patrick Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow). Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later.
Later,
Dean
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
This message and any attachments are intended only for the use of the
addressee and
may contain information that is privileged and confidential. If the
reader of the
message is not the intended recipient or an authorized representative of
the
intended recipient, you are hereby notified that any dissemination of
this
communication is strictly prohibited. If you have received this
communication in
error, please notify us immediately by e-mail and delete the message and
any
attachments from your system.
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Posted by "Hiller, Dean (Contractor)" <de...@broadridge.com>.
Thanks of the info, missed that at the bottom of that page.
Dean
-----Original Message-----
From: Fox, Kevin M [mailto:kevin.fox@pnl.gov]
Sent: Wednesday, December 29, 2010 2:21 PM
To: Hiller, Dean (Contractor); general@hadoop.apache.org; Patrick
Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
http://wiki.apache.org/hadoop/MountableHDFS
Under Known Issues:
2. Writes are approximately 33% slower than the DFSClient. TBD how to
optimize this. see: HADOOP-3805 - try using -obig_writes if on a
>2.6.26 kernel, should perform much better since bigger writes implies
less context switching.
3. Reads are ~20-30% slower even with the read buffering.
Sounds like just pushing it in would be better.
Thanks,
Kevin
-----Original Message-----
From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]
Sent: Wednesday, December 29, 2010 1:16 PM
To: general@hadoop.apache.org; Fox, Kevin M; Patrick Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow). Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later.
Later,
Dean
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
This message and any attachments are intended only for the use of the
addressee and
may contain information that is privileged and confidential. If the
reader of the
message is not the intended recipient or an authorized representative of
the
intended recipient, you are hereby notified that any dissemination of
this
communication is strictly prohibited. If you have received this
communication in
error, please notify us immediately by e-mail and delete the message and
any
attachments from your system.
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Fox, Kevin M" <ke...@pnl.gov>.
http://wiki.apache.org/hadoop/MountableHDFS
Under Known Issues:
2. Writes are approximately 33% slower than the DFSClient. TBD how to optimize this. see: HADOOP-3805 - try using -obig_writes if on a >2.6.26 kernel, should perform much better since bigger writes implies less context switching.
3. Reads are ~20-30% slower even with the read buffering.
Sounds like just pushing it in would be better.
Thanks,
Kevin
-----Original Message-----
From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]
Sent: Wednesday, December 29, 2010 1:16 PM
To: general@hadoop.apache.org; Fox, Kevin M; Patrick Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow). Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later.
Later,
Dean
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Ted Dunning <td...@maprtech.com>.
The problem there is that HDFS isn't a first class file system. That means
that the nice and easy ways of mounting will lead to problems (notably NFS
which maintains no state will require random write capabilities).
On Wed, Dec 29, 2010 at 1:16 PM, Hiller, Dean (Contractor) <
dean.hiller@broadridge.com> wrote:
> I wonder if having linux mount hdfs would help here so as people put the
> file on your linux /hdfs directory, it was actually writing to hdfs and
> not linux ;) (yeah, you still have that one machine bottle neck as the
> files come in unless that can be clustered too somehow). Just google
> mounting hdfs from linux....something that sounds pretty cool that we
> may be using later.
>
> Later,
> Dean
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 5:05 PM
> To: Fox, Kevin M; Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
> Taylor, Ronald C
> Subject: RE: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
>
> Hi Kevin,
>
> So - from what Patrick and Ted are saying it sounds like we want the
> best way to parallelize a source-based push, rather than doing a
> parallelized pull through a MapReduce program. And I see that what you
> ask about below is on parallelizing a push, so we are on the same page.
> Ron
>
> -----Original Message-----
> From: Fox, Kevin M
> Sent: Tuesday, December 28, 2010 3:39 PM
> To: Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
> Brown, David M JR
> Subject: Re: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
> On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> > Ron,
> >
> >
> > While MapReduce can help to parallelize the load effort, your likely
> > bottleneck is the source system (where the files come from). If the
> > files are coming from a single server, then parallelizing the load
> > won't gain you much past a certain point. You have to figure in how
> > fast you can read the file(s) off disk(s) and push the bits through
> > your network and finally onto HDFS.
> >
> >
> > The best scenario is if you can parallelize the reads and have a fat
> > network pipe (10GbE or more) going into your Hadoop cluster.
>
>
> We have a way to parallelize a push from the archive storage cluster to
> the hadoop storage cluster.
>
> Is there a way to target a particular storage node with a push into the
> hadoop file system? The hadoop cluster nodes are 1gig attached to its
> core switch and we have a 10 gig uplink to the core from the storage
> archive. Say, we have 4 nodes in each storage cluster (we have more,
> just a simplified example):
>
> a0 --\ /-- h0
> a1 --+ +-- h1
> a2 --+ (A switch) -10gige- (h switch) +-- h2
> a3 --/ \-- h3
>
> I want to be able to have a0 talk to h0 and not have h0 decide the data
> belongs on h3, slowing down a3's ability to write data into h3, greatly
> reducing bandwidth.
>
> Thanks,
> Kevin
>
> >
> >
> > Regards,
> >
> >
> > - Patrick
> >
> > On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> > <ro...@pnl.gov> wrote:
> >
> > Folks,
> >
> > We plan on uploading large amounts of data on a regular basis
> > onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> > Figure eventually on the order of multiple terabytes per week.
> > So - we are concerned about doing the uploads themselves as
> > fast as possible from our native Linux file system into HDFS.
> > Figure files will be in, roughly, the 1 to 300 GB range.
> >
> > Off the top of my head, I'm thinking that doing this in
> > parallel using a Java MapReduce program would work fastest. So
> > my idea would be to have a file listing all the data files
> > (full paths) to be uploaded, one per line, and then use that
> > listing file as input to a MapReduce program.
> >
> > Each Mapper would then upload one of the data files (using
> > "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> > all the other Mappers, with the Mappers operating on all the
> > nodes of the cluster, spreading out the file upload across the
> > nodes.
> >
> > Does that sound like a wise way to approach this? Are there
> > better methods? Anything else out there for doing automated
> > upload in parallel? We would very much appreciate advice in
> > this area, since we believe upload speed might become a
> > bottleneck.
> >
> > - Ron Taylor
> >
> > ___________________________________________
> > Ronald Taylor, Ph.D.
> > Computational Biology & Bioinformatics Group
> >
> > Pacific Northwest National Laboratory
> > 902 Battelle Boulevard
> > P.O. Box 999, Mail Stop J4-33
> > Richland, WA 99352 USA
> > Office: 509-372-6568
> > Email: ronald.taylor@pnl.gov
> >
> >
> >
> >
>
>
> This message and any attachments are intended only for the use of the
> addressee and
> may contain information that is privileged and confidential. If the reader
> of the
> message is not the intended recipient or an authorized representative of
> the
> intended recipient, you are hereby notified that any dissemination of this
> communication is strictly prohibited. If you have received this
> communication in
> error, please notify us immediately by e-mail and delete the message and
> any
> attachments from your system.
>
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Ted Dunning <td...@maprtech.com>.
The problem there is that HDFS isn't a first class file system. That means
that the nice and easy ways of mounting will lead to problems (notably NFS
which maintains no state will require random write capabilities).
On Wed, Dec 29, 2010 at 1:16 PM, Hiller, Dean (Contractor) <
dean.hiller@broadridge.com> wrote:
> I wonder if having linux mount hdfs would help here so as people put the
> file on your linux /hdfs directory, it was actually writing to hdfs and
> not linux ;) (yeah, you still have that one machine bottle neck as the
> files come in unless that can be clustered too somehow). Just google
> mounting hdfs from linux....something that sounds pretty cool that we
> may be using later.
>
> Later,
> Dean
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 5:05 PM
> To: Fox, Kevin M; Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
> Taylor, Ronald C
> Subject: RE: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
>
> Hi Kevin,
>
> So - from what Patrick and Ted are saying it sounds like we want the
> best way to parallelize a source-based push, rather than doing a
> parallelized pull through a MapReduce program. And I see that what you
> ask about below is on parallelizing a push, so we are on the same page.
> Ron
>
> -----Original Message-----
> From: Fox, Kevin M
> Sent: Tuesday, December 28, 2010 3:39 PM
> To: Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
> Brown, David M JR
> Subject: Re: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
> On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> > Ron,
> >
> >
> > While MapReduce can help to parallelize the load effort, your likely
> > bottleneck is the source system (where the files come from). If the
> > files are coming from a single server, then parallelizing the load
> > won't gain you much past a certain point. You have to figure in how
> > fast you can read the file(s) off disk(s) and push the bits through
> > your network and finally onto HDFS.
> >
> >
> > The best scenario is if you can parallelize the reads and have a fat
> > network pipe (10GbE or more) going into your Hadoop cluster.
>
>
> We have a way to parallelize a push from the archive storage cluster to
> the hadoop storage cluster.
>
> Is there a way to target a particular storage node with a push into the
> hadoop file system? The hadoop cluster nodes are 1gig attached to its
> core switch and we have a 10 gig uplink to the core from the storage
> archive. Say, we have 4 nodes in each storage cluster (we have more,
> just a simplified example):
>
> a0 --\ /-- h0
> a1 --+ +-- h1
> a2 --+ (A switch) -10gige- (h switch) +-- h2
> a3 --/ \-- h3
>
> I want to be able to have a0 talk to h0 and not have h0 decide the data
> belongs on h3, slowing down a3's ability to write data into h3, greatly
> reducing bandwidth.
>
> Thanks,
> Kevin
>
> >
> >
> > Regards,
> >
> >
> > - Patrick
> >
> > On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> > <ro...@pnl.gov> wrote:
> >
> > Folks,
> >
> > We plan on uploading large amounts of data on a regular basis
> > onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> > Figure eventually on the order of multiple terabytes per week.
> > So - we are concerned about doing the uploads themselves as
> > fast as possible from our native Linux file system into HDFS.
> > Figure files will be in, roughly, the 1 to 300 GB range.
> >
> > Off the top of my head, I'm thinking that doing this in
> > parallel using a Java MapReduce program would work fastest. So
> > my idea would be to have a file listing all the data files
> > (full paths) to be uploaded, one per line, and then use that
> > listing file as input to a MapReduce program.
> >
> > Each Mapper would then upload one of the data files (using
> > "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> > all the other Mappers, with the Mappers operating on all the
> > nodes of the cluster, spreading out the file upload across the
> > nodes.
> >
> > Does that sound like a wise way to approach this? Are there
> > better methods? Anything else out there for doing automated
> > upload in parallel? We would very much appreciate advice in
> > this area, since we believe upload speed might become a
> > bottleneck.
> >
> > - Ron Taylor
> >
> > ___________________________________________
> > Ronald Taylor, Ph.D.
> > Computational Biology & Bioinformatics Group
> >
> > Pacific Northwest National Laboratory
> > 902 Battelle Boulevard
> > P.O. Box 999, Mail Stop J4-33
> > Richland, WA 99352 USA
> > Office: 509-372-6568
> > Email: ronald.taylor@pnl.gov
> >
> >
> >
> >
>
>
> This message and any attachments are intended only for the use of the
> addressee and
> may contain information that is privileged and confidential. If the reader
> of the
> message is not the intended recipient or an authorized representative of
> the
> intended recipient, you are hereby notified that any dissemination of this
> communication is strictly prohibited. If you have received this
> communication in
> error, please notify us immediately by e-mail and delete the message and
> any
> attachments from your system.
>
>
RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Posted by "Hiller, Dean (Contractor)" <de...@broadridge.com>.
I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow). Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later.
Later,
Dean
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Kevin Fox <Ke...@pnl.gov>.
On Tue, 2010-12-28 at 16:04 -0800, Taylor, Ronald C wrote:
> Hi Kevin,
>
> So - from what Patrick and Ted are saying it sounds like we want the best way to parallelize a source-based push, rather than doing a parallelized pull through a MapReduce program. And I see that what you ask about below is on parallelizing a push, so we are on the same page.
> Ron
Hi Ron,
I think there are merits for both approaches, a depending on the type of
data it is. If you only ever need a subset of the data for reprocessing,
map/reducing then transfer might be better. If you want all the raw data
of a particular set, push transfer may make more sense.
So we'd need solutions to both:
*parallel map reduce on non hdfs storage cluster, push to hdfs
reprocessing cluster.
*parallel push data from non hdfs storage cluster, into hdfs
reprocessing cluster.
Thanks,
Kevin
>
> -----Original Message-----
> From: Fox, Kevin M
> Sent: Tuesday, December 28, 2010 3:39 PM
> To: Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C; Brown, David M JR
> Subject: Re: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
>
> On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> > Ron,
> >
> >
> > While MapReduce can help to parallelize the load effort, your likely
> > bottleneck is the source system (where the files come from). If the
> > files are coming from a single server, then parallelizing the load
> > won't gain you much past a certain point. You have to figure in how
> > fast you can read the file(s) off disk(s) and push the bits through
> > your network and finally onto HDFS.
> >
> >
> > The best scenario is if you can parallelize the reads and have a fat
> > network pipe (10GbE or more) going into your Hadoop cluster.
>
>
> We have a way to parallelize a push from the archive storage cluster to the hadoop storage cluster.
>
> Is there a way to target a particular storage node with a push into the hadoop file system? The hadoop cluster nodes are 1gig attached to its core switch and we have a 10 gig uplink to the core from the storage archive. Say, we have 4 nodes in each storage cluster (we have more, just a simplified example):
>
> a0 --\ /-- h0
> a1 --+ +-- h1
> a2 --+ (A switch) -10gige- (h switch) +-- h2
> a3 --/ \-- h3
>
> I want to be able to have a0 talk to h0 and not have h0 decide the data belongs on h3, slowing down a3's ability to write data into h3, greatly reducing bandwidth.
>
> Thanks,
> Kevin
>
> >
> >
> > Regards,
> >
> >
> > - Patrick
> >
> > On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> > <ro...@pnl.gov> wrote:
> >
> > Folks,
> >
> > We plan on uploading large amounts of data on a regular basis
> > onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> > Figure eventually on the order of multiple terabytes per week.
> > So - we are concerned about doing the uploads themselves as
> > fast as possible from our native Linux file system into HDFS.
> > Figure files will be in, roughly, the 1 to 300 GB range.
> >
> > Off the top of my head, I'm thinking that doing this in
> > parallel using a Java MapReduce program would work fastest. So
> > my idea would be to have a file listing all the data files
> > (full paths) to be uploaded, one per line, and then use that
> > listing file as input to a MapReduce program.
> >
> > Each Mapper would then upload one of the data files (using
> > "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> > all the other Mappers, with the Mappers operating on all the
> > nodes of the cluster, spreading out the file upload across the
> > nodes.
> >
> > Does that sound like a wise way to approach this? Are there
> > better methods? Anything else out there for doing automated
> > upload in parallel? We would very much appreciate advice in
> > this area, since we believe upload speed might become a
> > bottleneck.
> >
> > - Ron Taylor
> >
> > ___________________________________________
> > Ronald Taylor, Ph.D.
> > Computational Biology & Bioinformatics Group
> >
> > Pacific Northwest National Laboratory
> > 902 Battelle Boulevard
> > P.O. Box 999, Mail Stop J4-33
> > Richland, WA 99352 USA
> > Office: 509-372-6568
> > Email: ronald.taylor@pnl.gov
> >
> >
> >
> >
>
>
RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Posted by "Hiller, Dean (Contractor)" <de...@broadridge.com>.
I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow). Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later.
Later,
Dean
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Taylor, Ronald C" <ro...@pnl.gov>.
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the best way to parallelize a source-based push, rather than doing a parallelized pull through a MapReduce program. And I see that what you ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C; Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the hadoop file system? The hadoop cluster nodes are 1gig attached to its core switch and we have a 10 gig uplink to the core from the storage archive. Say, we have 4 nodes in each storage cluster (we have more, just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data belongs on h3, slowing down a3's ability to write data into h3, greatly reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Taylor, Ronald C" <ro...@pnl.gov>.
Hi Kevin,
So - from what Patrick and Ted are saying it sounds like we want the best way to parallelize a source-based push, rather than doing a parallelized pull through a MapReduce program. And I see that what you ask about below is on parallelizing a push, so we are on the same page.
Ron
-----Original Message-----
From: Fox, Kevin M
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C; Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the hadoop file system? The hadoop cluster nodes are 1gig attached to its core switch and we have a 10 gig uplink to the core from the storage archive. Say, we have 4 nodes in each storage cluster (we have more, just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data belongs on h3, slowing down a3's ability to write data into h3, greatly reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Kevin Fox <Ke...@pnl.gov>.
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Taylor, Ronald C" <ro...@pnl.gov>.
Patrick,
Thanks for the info (and quick reply). I want to make sure I understand: Presuming that the data files are coming off a set of disk drives attached to a single Linux file server, you say I need two things to optimize the transfer:
1) a fat network pipe
2) some way of parallelizing the reads
So - I will check into network hardware, in regard to (1). But for (2), is the MapReduce method that I was think of, a way that uses "hadoop fs -copyFromLocal" in each Mapper, a good way to go at the destination end? I believe that you were saying that it is indeed OK, but I want to double-check, since this will be a critical piece of our work flow.
Ron
________________________________
From: patrickangeles@gmail.com [mailto:patrickangeles@gmail.com] On Behalf Of Patrick Angeles
Sent: Tuesday, December 28, 2010 2:27 PM
To: general@hadoop.apache.org
Cc: user@hbase.apache.org; Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Ron,
While MapReduce can help to parallelize the load effort, your likely bottleneck is the source system (where the files come from). If the files are coming from a single server, then parallelizing the load won't gain you much past a certain point. You have to figure in how fast you can read the file(s) off disk(s) and push the bits through your network and finally onto HDFS.
The best scenario is if you can parallelize the reads and have a fat network pipe (10GbE or more) going into your Hadoop cluster.
Regards,
- Patrick
On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C <ro...@pnl.gov>> wrote:
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov<ma...@pnl.gov>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Kevin Fox <Ke...@pnl.gov>.
On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>
>
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
>
>
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
>
We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.
Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):
a0 --\ /-- h0
a1 --+ +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/ \-- h3
I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.
Thanks,
Kevin
>
> Regards,
>
>
> - Patrick
>
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ro...@pnl.gov> wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis
> onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> Figure eventually on the order of multiple terabytes per week.
> So - we are concerned about doing the uploads themselves as
> fast as possible from our native Linux file system into HDFS.
> Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in
> parallel using a Java MapReduce program would work fastest. So
> my idea would be to have a file listing all the data files
> (full paths) to be uploaded, one per line, and then use that
> listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using
> "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> all the other Mappers, with the Mappers operating on all the
> nodes of the cluster, spreading out the file upload across the
> nodes.
>
> Does that sound like a wise way to approach this? Are there
> better methods? Anything else out there for doing automated
> upload in parallel? We would very much appreciate advice in
> this area, since we believe upload speed might become a
> bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Patrick Angeles <pa...@cloudera.com>.
Ron,
While MapReduce can help to parallelize the load effort, your likely
bottleneck is the source system (where the files come from). If the files
are coming from a single server, then parallelizing the load won't gain you
much past a certain point. You have to figure in how fast you can read the
file(s) off disk(s) and push the bits through your network and finally onto
HDFS.
The best scenario is if you can parallelize the reads and have a fat network
pipe (10GbE or more) going into your Hadoop cluster.
Regards,
- Patrick
On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C <ro...@pnl.gov>wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop
> cluster, with Hbase operating on top of Hadoop. Figure eventually on the
> order of multiple terabytes per week. So - we are concerned about doing the
> uploads themselves as fast as possible from our native Linux file system
> into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a
> Java MapReduce program would work fastest. So my idea would be to have a
> file listing all the data files (full paths) to be uploaded, one per line,
> and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs
> -copyFromLocal <source> <dest>") in parallel with all the other Mappers,
> with the Mappers operating on all the nodes of the cluster, spreading out
> the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods?
> Anything else out there for doing automated upload in parallel? We would
> very much appreciate advice in this area, since we believe upload speed
> might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Kevin Fox <Ke...@pnl.gov>.
As I understand it, and please correct me if I'm wrong, a map/reduce job
has an instance of a FileSystem object on either side. One that the data
is read out of on the map side, and one the data is fed into on the
reduce side.
Can't you run the map reduce job on the storage cluster that stores the
archival data, feeding the map side with
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/RawLocalFileSystem.html
from the mounted, posix parallel file system,
and feed it into the hadoop cluster on the reduce side. That would mean,
the network in the middle would only see the reduced data set cross the
wire and you could parallelize the data reduction as close to the
archive as possible.
Thanks,
Kevin
On Tue, 2010-12-28 at 14:04 -0800, Taylor, Ronald C wrote:
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Taylor, Ronald C" <ro...@pnl.gov>.
Hi Dave,
Thanks for the suggestions. Glad to hear from a fellow DOE national lab person!
We are just starting to explore all this here at Pacific Northwest Nat Lab, and what will be going into Hbase and what will be left as files in HDFS is an open question, to be empirically determined over the coming year. It will depend on upon what instrument data gets put in, how the users want to analyze the data, what turns out to be practical for future growth and maintenance, etc. My lab colleagues Kevin Fox and David Brown have a lot more experience handling massive amount of data - they are already handling hundreds of TBs in the archive cluster for EMSL, our national user facility (lots of mass spec, NMR, microscopy, and next gen sequencing machines for biology and chemistry, as you may already know). And they have much better grip on the hardware and OS side of things. So I imagine you & the list will be hearing directly from them fairly often as questions arise.
Ron
-----Original Message-----
From: Buttler, David [mailto:buttler1@llnl.gov]
Sent: Monday, January 03, 2011 12:21 PM
To: user@hbase.apache.org; 'general@hadoop.apache.org'
Cc: Fox, Kevin M; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Hi Ron,
Loading into HDFS and HBase are two different issues.
HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help. You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first. If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task. Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
Dave
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 2:05 PM
To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Buttler, David" <bu...@llnl.gov>.
Right, I should have realized that you guys would be using a good parallel file system. In that case a M/R job will be great for moving the data -- as long as you don't overload the network. And if you are going to have the data end up in HBase you may just write a map job to directly insert into hbase, either through standard inserts or bulk loads (which should be 10x faster).
If you have time to play, it might be interesting to see how hbase runs over your current file system. Just use the file://<path to shared directory> instead of the hdfs:// url in the hbase-site.xml file. I would stress test that first with both hbase and m/r jobs just to make sure that it behaves well, but there would be many people who would love to hear about your experience here. I for one would like to know if I can run hbase over luster since we love luster around here.
Dave
-----Original Message-----
From: Kevin Fox [mailto:Kevin.Fox@pnl.gov]
Sent: Monday, January 03, 2011 2:58 PM
To: Buttler, David
Cc: user@hbase.apache.org; 'general@hadoop.apache.org'; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
On Mon, 2011-01-03 at 12:20 -0800, Buttler, David wrote:
> Hi Ron,
> Loading into HDFS and HBase are two different issues.
>
> HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help.
Its not nfs. Its a parallel file system.
> You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first.
Unlikely in this case. We're in the unusual position of our archive
cluster being faster then our hadoop cluster.
> If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task.
If the same files are "local" on each node, does it matter? Shouldn't
the map jobs all be scheduled in a way as to spread out the load?
Thanks,
Kevin
> Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
>
> HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
>
> Dave
>
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 2:05 PM
> To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
> Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
> Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
>
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Buttler, David" <bu...@llnl.gov>.
Right, I should have realized that you guys would be using a good parallel file system. In that case a M/R job will be great for moving the data -- as long as you don't overload the network. And if you are going to have the data end up in HBase you may just write a map job to directly insert into hbase, either through standard inserts or bulk loads (which should be 10x faster).
If you have time to play, it might be interesting to see how hbase runs over your current file system. Just use the file://<path to shared directory> instead of the hdfs:// url in the hbase-site.xml file. I would stress test that first with both hbase and m/r jobs just to make sure that it behaves well, but there would be many people who would love to hear about your experience here. I for one would like to know if I can run hbase over luster since we love luster around here.
Dave
-----Original Message-----
From: Kevin Fox [mailto:Kevin.Fox@pnl.gov]
Sent: Monday, January 03, 2011 2:58 PM
To: Buttler, David
Cc: user@hbase.apache.org; 'general@hadoop.apache.org'; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
On Mon, 2011-01-03 at 12:20 -0800, Buttler, David wrote:
> Hi Ron,
> Loading into HDFS and HBase are two different issues.
>
> HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help.
Its not nfs. Its a parallel file system.
> You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first.
Unlikely in this case. We're in the unusual position of our archive
cluster being faster then our hadoop cluster.
> If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task.
If the same files are "local" on each node, does it matter? Shouldn't
the map jobs all be scheduled in a way as to spread out the load?
Thanks,
Kevin
> Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
>
> HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
>
> Dave
>
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 2:05 PM
> To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
> Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
> Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
>
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Kevin Fox <Ke...@pnl.gov>.
On Mon, 2011-01-03 at 12:20 -0800, Buttler, David wrote:
> Hi Ron,
> Loading into HDFS and HBase are two different issues.
>
> HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help.
Its not nfs. Its a parallel file system.
> You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first.
Unlikely in this case. We're in the unusual position of our archive
cluster being faster then our hadoop cluster.
> If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task.
If the same files are "local" on each node, does it matter? Shouldn't
the map jobs all be scheduled in a way as to spread out the load?
Thanks,
Kevin
> Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
>
> HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
>
> Dave
>
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 2:05 PM
> To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
> Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
> Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
>
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Taylor, Ronald C" <ro...@pnl.gov>.
Hi Dave,
Thanks for the suggestions. Glad to hear from a fellow DOE national lab person!
We are just starting to explore all this here at Pacific Northwest Nat Lab, and what will be going into Hbase and what will be left as files in HDFS is an open question, to be empirically determined over the coming year. It will depend on upon what instrument data gets put in, how the users want to analyze the data, what turns out to be practical for future growth and maintenance, etc. My lab colleagues Kevin Fox and David Brown have a lot more experience handling massive amount of data - they are already handling hundreds of TBs in the archive cluster for EMSL, our national user facility (lots of mass spec, NMR, microscopy, and next gen sequencing machines for biology and chemistry, as you may already know). And they have much better grip on the hardware and OS side of things. So I imagine you & the list will be hearing directly from them fairly often as questions arise.
Ron
-----Original Message-----
From: Buttler, David [mailto:buttler1@llnl.gov]
Sent: Monday, January 03, 2011 12:21 PM
To: user@hbase.apache.org; 'general@hadoop.apache.org'
Cc: Fox, Kevin M; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Hi Ron,
Loading into HDFS and HBase are two different issues.
HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help. You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first. If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task. Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
Dave
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 2:05 PM
To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Buttler, David" <bu...@llnl.gov>.
Hi Ron,
Loading into HDFS and HBase are two different issues.
HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help. You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first. If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task. Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
Dave
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 2:05 PM
To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov
Re: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by Patrick Angeles <pa...@cloudera.com>.
Ron,
While MapReduce can help to parallelize the load effort, your likely
bottleneck is the source system (where the files come from). If the files
are coming from a single server, then parallelizing the load won't gain you
much past a certain point. You have to figure in how fast you can read the
file(s) off disk(s) and push the bits through your network and finally onto
HDFS.
The best scenario is if you can parallelize the reads and have a fat network
pipe (10GbE or more) going into your Hadoop cluster.
Regards,
- Patrick
On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C <ro...@pnl.gov>wrote:
>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop
> cluster, with Hbase operating on top of Hadoop. Figure eventually on the
> order of multiple terabytes per week. So - we are concerned about doing the
> uploads themselves as fast as possible from our native Linux file system
> into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a
> Java MapReduce program would work fastest. So my idea would be to have a
> file listing all the data files (full paths) to be uploaded, one per line,
> and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs
> -copyFromLocal <source> <dest>") in parallel with all the other Mappers,
> with the Mappers operating on all the nodes of the cluster, spreading out
> the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods?
> Anything else out there for doing automated upload in parallel? We would
> very much appreciate advice in this area, since we believe upload speed
> might become a bottleneck.
>
> - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA 99352 USA
> Office: 509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>
RE: What is the fastest way to get a large amount of data into the
Hadoop HDFS file system (or Hbase)?
Posted by "Buttler, David" <bu...@llnl.gov>.
Hi Ron,
Loading into HDFS and HBase are two different issues.
HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help. You have two sources of bottlenecks: the nfs file system and the HDFS file system. In your parallel example, you will likely saturate your nfs file system first. If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task. Unless all of the machines have files in the same directory and you are just going to look in that directory to upload. Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase. You can try to do this and see if direct inserts into hbase are fast enough for your use case. But, if you are going to TBs/week then you will likely want to investigate the bulk load features. I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately. E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs. That should give you a feeling for how much space you will need for your expected data load.
Dave
-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
Sent: Tuesday, December 28, 2010 2:05 PM
To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Folks,
We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program.
Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
- Ron Taylor
___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA 99352 USA
Office: 509-372-6568
Email: ronald.taylor@pnl.gov