You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Andrzej Jan Taramina <an...@chaeron.com> on 2009/09/14 15:40:39 UTC
Processing a large quantity of smaller XML files?
I'm new to Hadoop, so pardon the potentially dumb question....
I've gathered, from much research, that Hadoop is not always a good choice when you need to process a whack of smaller
files, which is what we need to do.
More specifically, we need to start by processing about 250K XML files, each of which is in the 50K - 2M range, with an
average size of 100K bytes. The processing we need to do on each file is pretty CPU-intensive, with a lot of pattern
matching. What we need to do would fall nicely into the Map/Reduce paradigm. Over time, the volume of files will grow
by an order of magnitude into the range of millions of files, hence the desire to use a mapred distributed cluster to do
the analysis we need.
Normally, one could just concatenate the XML files into bigger input files. Unfortunately, one of our constrains is
that a certain percentage of these XML files will change every night, and so we need to be able to update the Hadoop
data store (HDFS perhaps) on a regular basis. This would be difficult if the files are all concatenated.
The XML data originally comes from a number of XML databases.
Any advice/suggestions on the best way to structure our data storage of all the XML files so that Hadoop would run
efficiently and we could thus use Map/Reduce on a Hadoop cluster, yet still conveniently update the changed files on a
nightly basis?
Much appreciated!
--
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com
Re: Processing a large quantity of smaller XML files?
Posted by brien colwell <xc...@gmail.com>.
We have a similar setup of many smaller files. We found that using a
combined file input (64mb of files are bundled as input to a single map
job) in conjunction with compressed files on the HDFS much reduced the
overhead of the map jobs. I haven't compared distributing ZIP/TAR files
on the HDFS versus using the combined file input ... I assume the
combined file input is efficient but have not measured versus explicitly
bundling the input files.
Andrzej Jan Taramina wrote:
> I'm new to Hadoop, so pardon the potentially dumb question....
>
> I've gathered, from much research, that Hadoop is not always a good choice when you need to process a whack of smaller
> files, which is what we need to do.
>
> More specifically, we need to start by processing about 250K XML files, each of which is in the 50K - 2M range, with an
> average size of 100K bytes. The processing we need to do on each file is pretty CPU-intensive, with a lot of pattern
> matching. What we need to do would fall nicely into the Map/Reduce paradigm. Over time, the volume of files will grow
> by an order of magnitude into the range of millions of files, hence the desire to use a mapred distributed cluster to do
> the analysis we need.
>
> Normally, one could just concatenate the XML files into bigger input files. Unfortunately, one of our constrains is
> that a certain percentage of these XML files will change every night, and so we need to be able to update the Hadoop
> data store (HDFS perhaps) on a regular basis. This would be difficult if the files are all concatenated.
>
> The XML data originally comes from a number of XML databases.
>
> Any advice/suggestions on the best way to structure our data storage of all the XML files so that Hadoop would run
> efficiently and we could thus use Map/Reduce on a Hadoop cluster, yet still conveniently update the changed files on a
> nightly basis?
>
> Much appreciated!
>
>
Re: Re: Processing a large quantity of smaller XML files?
Posted by Ted Dunning <te...@gmail.com>.
If you have a unique id for each file (as you must if you are updating
them), then I think you would find it pretty surprising how fast you can
merge your old archive file and the updated/added versions.
This is a special case of the TB update puzzle. If you take as given that
you have a conventional disk with 100MB/s transfer and 10ms seek and
rotation times, and supposing that you want to update 1% of the 100B records
in a TB database, you will find that simply copying the entire database,
inserting the updated records into the copy at the right places is about
100x faster than naive random accesses.
Moreover, in your case, you will be able to parallelize the update step
which should make it even faster. If your updates are available in sorted
order, then you can do this with a map-side merge and no reduce.
You should be able to sustain about 200-500MB/s read speed for every 10
spindles that your Hadoop lives on. Assuming that your cluster is 20-50
nodes, you should be able to update your database at about 20 minutes or
less per TB of data. Is that really too slow?
The cheap alternative is to just keep each nights updates in a separate
archive and keep a separate file with pointers to which archive the latest
version is in. Your map program would consult the separate file as each
file is read to determine if it is looking at the most recent and only use
it if it is recent. Occasional merges would be in order to make sure most
of the files you scan are relatively large. This is essentially what hbase
would be doing, but is stripped to the essentials.
On Thu, Sep 17, 2009 at 5:59 PM, Andrzej Jan Taramina
<an...@coalese.com>wrote:
> > The simplest thing you could do is to use the Hadoop ARchive format
> > (HAR) in a pre-processing step. The best thing you could do is to have
> > a pre-processing step based on sequence file (note: either Oozie or
> > Cascading are great workflow systems to help you out).
>
> That doesn't work, since some of our files are updated every night.
>
> > When you say "update" nightly, do you mean "add new files" or "update
> > existing files"? If you really mean changing existing files, HBase
> > might be good for you -
>
> We have to change existing files...and add some new ones as well. So HAR
> won't really cut it for us.
>
--
Ted Dunning, CTO
DeepDyve
Re: Re: Processing a large quantity of smaller XML files?
Posted by Andrzej Jan Taramina <an...@coalese.com>.
Brian (and others):
Great info...thanks!
> I would suggest looking into at Cloudera's blog posting about the "small
> files problem":
>
> http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/
Good link...muchos gracias.
> The simplest thing you could do is to use the Hadoop ARchive format
> (HAR) in a pre-processing step. The best thing you could do is to have
> a pre-processing step based on sequence file (note: either Oozie or
> Cascading are great workflow systems to help you out).
That doesn't work, since some of our files are updated every night.
> When you say "update" nightly, do you mean "add new files" or "update
> existing files"? If you really mean changing existing files, HBase
> might be good for you -
We have to change existing files...and add some new ones as well. So HAR won't really cut it for us.
> http://wiki.apache.org/hadoop/Hbase/HbaseArchitecture. HBase natively
> includes the concept of a timestamp, so you will be able to run against
> the the "latest version" or be able to specify a fixed version (in case
> if you want to repeat a night's analysis).
HBase sounds like the best approach for us, having digested the replies and also read the Cloudera link.
The question this raises is how to deploy Hadoop and HBase together?
What I would like to do is to have Hadoop data nodes also running HBase regionservers on the same machine if this is
feasible, that way when a Map/Reduce job runs, the data it will access should be local to the machine (eg. local HBase
region), at least in theory, right?
Is this doable/advisable? Anyone done this before...that is, having a Hadoop/HDFS data node running on the same machine
as a HBase regionserver, where a mapred job running on the Hadoop node will access local data on that machine? Config
advice to do this?
Thx!
> If you have separate archival storage for your data, you can also
> consider reloading every night; this might be scalable up to a few TB.
I would rather avoid that...since the data sources are remote in many cases, so aggregating a whole transmission every
night is probably not workable.
--
Andrzej Taramina
Coalese Corporation: Knowledge that Acts!
http://www.coalese.com
Re: Re: Processing a large quantity of smaller XML files?
Posted by Andrzej Jan Taramina <an...@chaeron.com>.
Brian (and others):
Great info...thanks!
> I would suggest looking into at Cloudera's blog posting about the "small
> files problem":
>
> http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/
Good link...muchos gracias.
> The simplest thing you could do is to use the Hadoop ARchive format
> (HAR) in a pre-processing step. The best thing you could do is to have
> a pre-processing step based on sequence file (note: either Oozie or
> Cascading are great workflow systems to help you out).
That doesn't work, since some of our files are updated every night.
> When you say "update" nightly, do you mean "add new files" or "update
> existing files"? If you really mean changing existing files, HBase
> might be good for you -
We have to change existing files...and add some new ones as well. So HAR won't really cut it for us.
> http://wiki.apache.org/hadoop/Hbase/HbaseArchitecture. HBase natively
> includes the concept of a timestamp, so you will be able to run against
> the the "latest version" or be able to specify a fixed version (in case
> if you want to repeat a night's analysis).
HBase sounds like the best approach for us, having digested the replies and also read the Cloudera link.
The question this raises is how to deploy Hadoop and HBase together?
What I would like to do is to have Hadoop data nodes also running HBase regionservers on the same machine if this is
feasible, that way when a Map/Reduce job runs, the data it will access should be local to the machine (eg. local HBase
region), at least in theory, right?
Is this doable/advisable? Anyone done this before...that is, having a Hadoop/HDFS data node running on the same machine
as a HBase regionserver, where a mapred job running on the Hadoop node will access local data on that machine? Config
advice to do this?
Thx!
> If you have separate archival storage for your data, you can also
> consider reloading every night; this might be scalable up to a few TB.
I would rather avoid that...since the data sources are remote in many cases, so aggregating a whole transmission every
night is probably not workable.
--
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com
Re: Processing a large quantity of smaller XML files?
Posted by Ryan Rawson <ry...@gmail.com>.
Random writes is kind of the area where HBase is great at. Like you
said, you cant just rewrite large sequence files every time you have a
change, so you would consider using a system like HBase to help with
that.
I'd say if you want some map-reduce capacity, hdfs, hbase you will
need 4 cpus per node minimum, and 8 ideal. At 4 cpus per node you
might not want to use more than 1 heavy duty map per node.
let us know if you need any other tips...
-ryan
On Wed, Sep 23, 2009 at 3:30 PM, stack <st...@duboce.net> wrote:
> On Wed, Sep 23, 2009 at 2:01 PM, Andrzej Jan Taramina
> <an...@chaeron.com>wrote:
>
>> I asked about the best way to process a large quantity of smaller XML files
>> using Hadoop mapred, on the main Hadoop
>> mailing list, and was advised that HBase would be a good alternative to
>> handle this.
>>
>> ...
>
>>
>> What I would like to do is to have Hadoop data nodes also running HBase
>> regionservers on the same machine if this is
>> feasible, so that when a Map/Reduce job runs, the data it needs to access
>> would ideally be local to the machine (eg.
>> local HBase region), at least in theory.
>
>
>> Is this doable/advisable? Anyone done this before...that is, having a
>> Hadoop/HDFS data node running on the same machine
>> as a HBase regionserver, where a mapred job running on the Hadoop node will
>> access local data on that machine?
>>
>
> This is probably the most common deploy pattern (and yes, MR will assign the
> task to the tasktracker that is running beside the regionserver hosting the
> region it is supposed to work on) . The common mistake people make is that
> they do this on underpowered machines. Check our mail archives for multiple
> instances of folks overwhelming their hbase/hdfs with mapreduce task
> children run amok with i/o and cpu starving hdfs and hbase (See mail with
> the subject "HBase, Bigtable, and storage engineering..."; its starts out
> with a characterization of this phenomeon). MapReduce is batch-orientated,
> generally idempotent with ten minute timeouts and on failure, tasks are
> retried; i.e. its sloppy. HBase is at the other extreme. So just be
> consious that a balance has to be struck between the contending processes.
> We can help with this if you have trouble figuring it yourself.
>
> Yours,
> St.Ack
>
Re: Processing a large quantity of smaller XML files?
Posted by stack <st...@duboce.net>.
On Wed, Sep 23, 2009 at 2:01 PM, Andrzej Jan Taramina
<an...@chaeron.com>wrote:
> I asked about the best way to process a large quantity of smaller XML files
> using Hadoop mapred, on the main Hadoop
> mailing list, and was advised that HBase would be a good alternative to
> handle this.
>
> ...
>
> What I would like to do is to have Hadoop data nodes also running HBase
> regionservers on the same machine if this is
> feasible, so that when a Map/Reduce job runs, the data it needs to access
> would ideally be local to the machine (eg.
> local HBase region), at least in theory.
> Is this doable/advisable? Anyone done this before...that is, having a
> Hadoop/HDFS data node running on the same machine
> as a HBase regionserver, where a mapred job running on the Hadoop node will
> access local data on that machine?
>
This is probably the most common deploy pattern (and yes, MR will assign the
task to the tasktracker that is running beside the regionserver hosting the
region it is supposed to work on) . The common mistake people make is that
they do this on underpowered machines. Check our mail archives for multiple
instances of folks overwhelming their hbase/hdfs with mapreduce task
children run amok with i/o and cpu starving hdfs and hbase (See mail with
the subject "HBase, Bigtable, and storage engineering..."; its starts out
with a characterization of this phenomeon). MapReduce is batch-orientated,
generally idempotent with ten minute timeouts and on failure, tasks are
retried; i.e. its sloppy. HBase is at the other extreme. So just be
consious that a balance has to be struck between the contending processes.
We can help with this if you have trouble figuring it yourself.
Yours,
St.Ack
Processing a large quantity of smaller XML files?
Posted by Andrzej Jan Taramina <an...@chaeron.com>.
I asked about the best way to process a large quantity of smaller XML files using Hadoop mapred, on the main Hadoop
mailing list, and was advised that HBase would be a good alternative to handle this.
More specifically, we need to start by processing about 250K XML files, each of which is in the 50K - 2M range, with an
average size of 100K bytes. The processing we need to do on each file is pretty CPU-intensive, with a lot of pattern
matching. What we need to do would fall nicely into the Map/Reduce paradigm. Over time, the volume of files will grow
by an order of magnitude into the range of millions of files, hence the desire to use a mapred distributed cluster to do
the analysis we need.
Normally, one could just concatenate the XML files into bigger input files. Unfortunately, one of our constrains is
that a certain percentage of these XML files will change every night, and so we need to be able to update the Hadoop
data store (HDFS perhaps) on a regular basis. This would be difficult if the files are all concatenated.
The XML data originally comes from a number of XML databases.
HBase sounds like it might be a good approach for us, having digested the replies on the Hadoop list and also having
read the Cloudera link (http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/)
The question this raises is how to deploy Hadoop and HBase together in an optimal manner? Being a newbie to both Hadoop
and HBase, I figured it would be a good idea to consult the group about this, thus trying to avoid making any mistakes
that are commonly known and get us to where we need to be quickly.
What I would like to do is to have Hadoop data nodes also running HBase regionservers on the same machine if this is
feasible, so that when a Map/Reduce job runs, the data it needs to access would ideally be local to the machine (eg.
local HBase region), at least in theory.
Is this doable/advisable? Anyone done this before...that is, having a Hadoop/HDFS data node running on the same machine
as a HBase regionserver, where a mapred job running on the Hadoop node will access local data on that machine?
Is there any info available on how to configure such a scenario?
Thx!
--
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com
Re: Processing a large quantity of smaller XML files?
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Sep 14, 2009, at 8:44 AM, Piotr Praczyk wrote:
> Hi
>
> Maybe you should consider using HBase instead of pure HDFS ?
> HDFS tends to have a big block size which would lead to a massive
> storage
> space loss.
Nope, this is not true at all. If you have 64MB blocks and write a
32KB file, the space physically used on disk is ... 32KB.
250K XML files is not insane, and your NN would be able to *easily*
handle it (10 million files in HDFS would require 4GB of RAM in the
NN). However, you've correctly noticed that because you *can* handle
it, it does not mean it would be *efficient*.
I would suggest looking into at Cloudera's blog posting about the
"small files problem":
http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/
The simplest thing you could do is to use the Hadoop ARchive format
(HAR) in a pre-processing step. The best thing you could do is to
have a pre-processing step based on sequence file (note: either Oozie
or Cascading are great workflow systems to help you out).
When you say "update" nightly, do you mean "add new files" or "update
existing files"? If you really mean changing existing files, HBase
might be good for you - http://wiki.apache.org/hadoop/Hbase/HbaseArchitecture
. HBase natively includes the concept of a timestamp, so you will be
able to run against the the "latest version" or be able to specify a
fixed version (in case if you want to repeat a night's analysis).
If you have separate archival storage for your data, you can also
consider reloading every night; this might be scalable up to a few TB.
Brian
> HBase runs on top of HDFS and would store many files in the same
> block yet allowing to modify them selectively.
>
> regards
> Piotr
>
> 2009/9/14 Andrzej Jan Taramina <an...@chaeron.com>
>
>> I'm new to Hadoop, so pardon the potentially dumb question....
>>
>> I've gathered, from much research, that Hadoop is not always a good
>> choice
>> when you need to process a whack of smaller
>> files, which is what we need to do.
>>
>> More specifically, we need to start by processing about 250K XML
>> files,
>> each of which is in the 50K - 2M range, with an
>> average size of 100K bytes. The processing we need to do on each
>> file is
>> pretty CPU-intensive, with a lot of pattern
>> matching. What we need to do would fall nicely into the Map/Reduce
>> paradigm. Over time, the volume of files will grow
>> by an order of magnitude into the range of millions of files, hence
>> the
>> desire to use a mapred distributed cluster to do
>> the analysis we need.
>>
>> Normally, one could just concatenate the XML files into bigger
>> input files.
>> Unfortunately, one of our constrains is
>> that a certain percentage of these XML files will change every
>> night, and
>> so we need to be able to update the Hadoop
>> data store (HDFS perhaps) on a regular basis. This would be
>> difficult if
>> the files are all concatenated.
>>
>> The XML data originally comes from a number of XML databases.
>>
>> Any advice/suggestions on the best way to structure our data
>> storage of all
>> the XML files so that Hadoop would run
>> efficiently and we could thus use Map/Reduce on a Hadoop cluster,
>> yet still
>> conveniently update the changed files on a
>> nightly basis?
>>
>> Much appreciated!
>>
>> --
>> Andrzej Taramina
>> Chaeron Corporation: Enterprise System Solutions
>> http://www.chaeron.com
>>
Re: Processing a large quantity of smaller XML files?
Posted by Piotr Praczyk <pi...@gmail.com>.
Hi
Maybe you should consider using HBase instead of pure HDFS ?
HDFS tends to have a big block size which would lead to a massive storage
space loss. HBase runs on top of HDFS and would store many files in the same
block yet allowing to modify them selectively.
regards
Piotr
2009/9/14 Andrzej Jan Taramina <an...@chaeron.com>
> I'm new to Hadoop, so pardon the potentially dumb question....
>
> I've gathered, from much research, that Hadoop is not always a good choice
> when you need to process a whack of smaller
> files, which is what we need to do.
>
> More specifically, we need to start by processing about 250K XML files,
> each of which is in the 50K - 2M range, with an
> average size of 100K bytes. The processing we need to do on each file is
> pretty CPU-intensive, with a lot of pattern
> matching. What we need to do would fall nicely into the Map/Reduce
> paradigm. Over time, the volume of files will grow
> by an order of magnitude into the range of millions of files, hence the
> desire to use a mapred distributed cluster to do
> the analysis we need.
>
> Normally, one could just concatenate the XML files into bigger input files.
> Unfortunately, one of our constrains is
> that a certain percentage of these XML files will change every night, and
> so we need to be able to update the Hadoop
> data store (HDFS perhaps) on a regular basis. This would be difficult if
> the files are all concatenated.
>
> The XML data originally comes from a number of XML databases.
>
> Any advice/suggestions on the best way to structure our data storage of all
> the XML files so that Hadoop would run
> efficiently and we could thus use Map/Reduce on a Hadoop cluster, yet still
> conveniently update the changed files on a
> nightly basis?
>
> Much appreciated!
>
> --
> Andrzej Taramina
> Chaeron Corporation: Enterprise System Solutions
> http://www.chaeron.com
>