You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Darren Govoni <da...@ontrenet.com> on 2009/01/21 14:07:37 UTC
Suitable for Hadoop?
Hi,
I have a task to process large quantities of files by converting them
into other formats. Each file is processed as a whole and converted to a
target format. Since there are 100's of GB of data I thought it suitable
for Hadoop, but the problem is, I don't think the files can be broken
apart and processed. For example, how would mapreduce work to convert a
Word Document to PDF if the file is reduced to blocks? I'm not sure
that's possible, or is it?
thanks for any advice
Darren
RE: Suitable for Hadoop?
Posted by Darren Govoni <da...@ontrenet.com>.
Richard,
Thanks for the suggestion. I actually am building an EC2 architecture
to facilitate this! I tried using a database to warehouse the files, and
then NFS but the connection load is too heavy. So I thought maybe HDFS
could be used just too mitigate the data access across all the
instances. I have a parallel processing architecture based on SQS
queues, but will consider a map process.
I have about 32 processes or so per machine in EC2 reading from SQS
queues for files to process, they could then efficiently get the files
from HDFS, yes? Without bottlenecking access to a database or NFS
server?
Well, i will test this direction and see too.
Thank you!
Darren
On Wed, 2009-01-21 at 09:41 -0500, Zak, Richard [USA] wrote:
> You can do that. I did a Map/Reduce job for about 6 GB of PDFs to
> concatenate them, and the New York times used Hadoop to process a few TB of
> PDFs.
>
> What I would do is this:
> - Use the iText library, a Java library for PDF manipulation (don't know
> what you would use for reading Word docs)
> - Don't use any Reducers
> - Have the input be a text file with the directory(ies) to process, since
> the mapper takes in file contents (and you don't want to read in one line of
> binary)
> - Have the map process all contents for that one given directory from the
> input text file
> - Break down the documents into more directories to go easier on the memory
> - Use Amazon's EC2, and the scripts in <hadoop_dir>/src/contrib/ec2/bin/
> (there is a script which passes environment variables to launched instances,
> modify the script to allow Hadoop to use more memory by setting the
> HADOOP_HEAPSIZE environment variable and having the variable properly
> passed)
>
> I realize this isn't the strong point of Map/Reduce or Hadoop, but it still
> uses the HDFS in a beneficial manner, and the distributed part is very
> helpful!
>
>
> Richard J. Zak
>
> -----Original Message-----
> From: Darren Govoni [mailto:darren@ontrenet.com]
> Sent: Wednesday, January 21, 2009 08:08
> To: core-user@hadoop.apache.org
> Subject: Suitable for Hadoop?
>
> Hi,
> I have a task to process large quantities of files by converting them into
> other formats. Each file is processed as a whole and converted to a target
> format. Since there are 100's of GB of data I thought it suitable for
> Hadoop, but the problem is, I don't think the files can be broken apart and
> processed. For example, how would mapreduce work to convert a Word Document
> to PDF if the file is reduced to blocks? I'm not sure that's possible, or is
> it?
>
> thanks for any advice
> Darren
>
RE: Suitable for Hadoop?
Posted by "Zak, Richard [USA]" <za...@bah.com>.
Darren- I would definitely use HDFS to get the data to all the instances.
I'm not sure about your 32 processes or SQS, but let me/us know what you
find.
Richard J. Zak
-----Original Message-----
From: Ricky Ho [mailto:rho@adobe.com]
Sent: Wednesday, January 21, 2009 15:00
To: core-user@hadoop.apache.org
Subject: RE: Suitable for Hadoop?
Jim, thanks for your explanation. But isn't isSplittable an option in
writing output rather than reading input ?
There are two phases.
1) Upload the data from local file to HDFS. Is there an option in the
"hadoop fs copy" to pack multiple small files in a single block and also not
splitting the files. So should "isSplittable" be applied here ?
2) The Mapper will start reading data using MultiFileInputFormat, which will
data in units of block size. Therefore a single mapper will get all the
files within its assigned block. Right ?
Rgds,
Ricky
-----Original Message-----
From: Jim Twensky [mailto:jim.twensky@gmail.com]
Sent: Wednesday, January 21, 2009 11:47 AM
To: core-user@hadoop.apache.org
Subject: Re: Suitable for Hadoop?
Ricky,
Hadoop was formerly optimized for large files, usually files of size larger
than one input split. However, there is an input format called
MultiFileInputFormat which can be used to utilize Hadoop to work efficiently
on smaller files. You can also set the isSplittable method of an input
format to "false" and ensure that a file is not split into pieces but rather
processed by only one mapper.
Jim
On Wed, Jan 21, 2009 at 9:14 AM, Ricky Ho <rh...@adobe.com> wrote:
> Hmmm ...
>
> From a space efficiency perspective, given HDFS (with large block
> size) is expecting large files, is Hadoop optimized for processing
> large number of small files ? Does each file take up at least 1 block
> ? or multiple files can sit on the same block.
>
> Rgds,
> Ricky
> -----Original Message-----
> From: Zak, Richard [USA] [mailto:zak_richard@bah.com]
> Sent: Wednesday, January 21, 2009 6:42 AM
> To: core-user@hadoop.apache.org
> Subject: RE: Suitable for Hadoop?
>
> You can do that. I did a Map/Reduce job for about 6 GB of PDFs to
> concatenate them, and the New York times used Hadoop to process a few
> TB of PDFs.
>
> What I would do is this:
> - Use the iText library, a Java library for PDF manipulation (don't
> know what you would use for reading Word docs)
> - Don't use any Reducers
> - Have the input be a text file with the directory(ies) to process,
> since the mapper takes in file contents (and you don't want to read in
> one line of
> binary)
> - Have the map process all contents for that one given directory from
> the input text file
> - Break down the documents into more directories to go easier on the
> memory
> - Use Amazon's EC2, and the scripts in
> <hadoop_dir>/src/contrib/ec2/bin/ (there is a script which passes
> environment variables to launched instances, modify the script to
> allow Hadoop to use more memory by setting the HADOOP_HEAPSIZE
> environment variable and having the variable properly
> passed)
>
> I realize this isn't the strong point of Map/Reduce or Hadoop, but it
> still uses the HDFS in a beneficial manner, and the distributed part
> is very helpful!
>
>
> Richard J. Zak
>
> -----Original Message-----
> From: Darren Govoni [mailto:darren@ontrenet.com]
> Sent: Wednesday, January 21, 2009 08:08
> To: core-user@hadoop.apache.org
> Subject: Suitable for Hadoop?
>
> Hi,
> I have a task to process large quantities of files by converting them
> into other formats. Each file is processed as a whole and converted to
> a target format. Since there are 100's of GB of data I thought it
> suitable for Hadoop, but the problem is, I don't think the files can
> be broken apart and processed. For example, how would mapreduce work
> to convert a Word Document to PDF if the file is reduced to blocks?
> I'm not sure that's possible, or is it?
>
> thanks for any advice
> Darren
>
>
RE: Suitable for Hadoop?
Posted by Ricky Ho <rh...@adobe.com>.
Jim, thanks for your explanation. But isn't isSplittable an option in writing output rather than reading input ?
There are two phases.
1) Upload the data from local file to HDFS. Is there an option in the "hadoop fs copy" to pack multiple small files in a single block and also not splitting the files. So should "isSplittable" be applied here ?
2) The Mapper will start reading data using MultiFileInputFormat, which will data in units of block size. Therefore a single mapper will get all the files within its assigned block. Right ?
Rgds,
Ricky
-----Original Message-----
From: Jim Twensky [mailto:jim.twensky@gmail.com]
Sent: Wednesday, January 21, 2009 11:47 AM
To: core-user@hadoop.apache.org
Subject: Re: Suitable for Hadoop?
Ricky,
Hadoop was formerly optimized for large files, usually files of size larger
than one input split. However, there is an input format called
MultiFileInputFormat which can be used to utilize Hadoop to work efficiently
on smaller files. You can also set the isSplittable method of an input
format to "false" and ensure that a file is not split into pieces but rather
processed by only one mapper.
Jim
On Wed, Jan 21, 2009 at 9:14 AM, Ricky Ho <rh...@adobe.com> wrote:
> Hmmm ...
>
> From a space efficiency perspective, given HDFS (with large block size) is
> expecting large files, is Hadoop optimized for processing large number of
> small files ? Does each file take up at least 1 block ? or multiple files
> can sit on the same block.
>
> Rgds,
> Ricky
> -----Original Message-----
> From: Zak, Richard [USA] [mailto:zak_richard@bah.com]
> Sent: Wednesday, January 21, 2009 6:42 AM
> To: core-user@hadoop.apache.org
> Subject: RE: Suitable for Hadoop?
>
> You can do that. I did a Map/Reduce job for about 6 GB of PDFs to
> concatenate them, and the New York times used Hadoop to process a few TB of
> PDFs.
>
> What I would do is this:
> - Use the iText library, a Java library for PDF manipulation (don't know
> what you would use for reading Word docs)
> - Don't use any Reducers
> - Have the input be a text file with the directory(ies) to process, since
> the mapper takes in file contents (and you don't want to read in one line
> of
> binary)
> - Have the map process all contents for that one given directory from the
> input text file
> - Break down the documents into more directories to go easier on the memory
> - Use Amazon's EC2, and the scripts in <hadoop_dir>/src/contrib/ec2/bin/
> (there is a script which passes environment variables to launched
> instances,
> modify the script to allow Hadoop to use more memory by setting the
> HADOOP_HEAPSIZE environment variable and having the variable properly
> passed)
>
> I realize this isn't the strong point of Map/Reduce or Hadoop, but it still
> uses the HDFS in a beneficial manner, and the distributed part is very
> helpful!
>
>
> Richard J. Zak
>
> -----Original Message-----
> From: Darren Govoni [mailto:darren@ontrenet.com]
> Sent: Wednesday, January 21, 2009 08:08
> To: core-user@hadoop.apache.org
> Subject: Suitable for Hadoop?
>
> Hi,
> I have a task to process large quantities of files by converting them into
> other formats. Each file is processed as a whole and converted to a target
> format. Since there are 100's of GB of data I thought it suitable for
> Hadoop, but the problem is, I don't think the files can be broken apart and
> processed. For example, how would mapreduce work to convert a Word Document
> to PDF if the file is reduced to blocks? I'm not sure that's possible, or
> is
> it?
>
> thanks for any advice
> Darren
>
>
Re: Suitable for Hadoop?
Posted by Jim Twensky <ji...@gmail.com>.
Ricky,
Hadoop was formerly optimized for large files, usually files of size larger
than one input split. However, there is an input format called
MultiFileInputFormat which can be used to utilize Hadoop to work efficiently
on smaller files. You can also set the isSplittable method of an input
format to "false" and ensure that a file is not split into pieces but rather
processed by only one mapper.
Jim
On Wed, Jan 21, 2009 at 9:14 AM, Ricky Ho <rh...@adobe.com> wrote:
> Hmmm ...
>
> From a space efficiency perspective, given HDFS (with large block size) is
> expecting large files, is Hadoop optimized for processing large number of
> small files ? Does each file take up at least 1 block ? or multiple files
> can sit on the same block.
>
> Rgds,
> Ricky
> -----Original Message-----
> From: Zak, Richard [USA] [mailto:zak_richard@bah.com]
> Sent: Wednesday, January 21, 2009 6:42 AM
> To: core-user@hadoop.apache.org
> Subject: RE: Suitable for Hadoop?
>
> You can do that. I did a Map/Reduce job for about 6 GB of PDFs to
> concatenate them, and the New York times used Hadoop to process a few TB of
> PDFs.
>
> What I would do is this:
> - Use the iText library, a Java library for PDF manipulation (don't know
> what you would use for reading Word docs)
> - Don't use any Reducers
> - Have the input be a text file with the directory(ies) to process, since
> the mapper takes in file contents (and you don't want to read in one line
> of
> binary)
> - Have the map process all contents for that one given directory from the
> input text file
> - Break down the documents into more directories to go easier on the memory
> - Use Amazon's EC2, and the scripts in <hadoop_dir>/src/contrib/ec2/bin/
> (there is a script which passes environment variables to launched
> instances,
> modify the script to allow Hadoop to use more memory by setting the
> HADOOP_HEAPSIZE environment variable and having the variable properly
> passed)
>
> I realize this isn't the strong point of Map/Reduce or Hadoop, but it still
> uses the HDFS in a beneficial manner, and the distributed part is very
> helpful!
>
>
> Richard J. Zak
>
> -----Original Message-----
> From: Darren Govoni [mailto:darren@ontrenet.com]
> Sent: Wednesday, January 21, 2009 08:08
> To: core-user@hadoop.apache.org
> Subject: Suitable for Hadoop?
>
> Hi,
> I have a task to process large quantities of files by converting them into
> other formats. Each file is processed as a whole and converted to a target
> format. Since there are 100's of GB of data I thought it suitable for
> Hadoop, but the problem is, I don't think the files can be broken apart and
> processed. For example, how would mapreduce work to convert a Word Document
> to PDF if the file is reduced to blocks? I'm not sure that's possible, or
> is
> it?
>
> thanks for any advice
> Darren
>
>
RE: Suitable for Hadoop?
Posted by Ricky Ho <rh...@adobe.com>.
Hmmm ...
>From a space efficiency perspective, given HDFS (with large block size) is expecting large files, is Hadoop optimized for processing large number of small files ? Does each file take up at least 1 block ? or multiple files can sit on the same block.
Rgds,
Ricky
-----Original Message-----
From: Zak, Richard [USA] [mailto:zak_richard@bah.com]
Sent: Wednesday, January 21, 2009 6:42 AM
To: core-user@hadoop.apache.org
Subject: RE: Suitable for Hadoop?
You can do that. I did a Map/Reduce job for about 6 GB of PDFs to
concatenate them, and the New York times used Hadoop to process a few TB of
PDFs.
What I would do is this:
- Use the iText library, a Java library for PDF manipulation (don't know
what you would use for reading Word docs)
- Don't use any Reducers
- Have the input be a text file with the directory(ies) to process, since
the mapper takes in file contents (and you don't want to read in one line of
binary)
- Have the map process all contents for that one given directory from the
input text file
- Break down the documents into more directories to go easier on the memory
- Use Amazon's EC2, and the scripts in <hadoop_dir>/src/contrib/ec2/bin/
(there is a script which passes environment variables to launched instances,
modify the script to allow Hadoop to use more memory by setting the
HADOOP_HEAPSIZE environment variable and having the variable properly
passed)
I realize this isn't the strong point of Map/Reduce or Hadoop, but it still
uses the HDFS in a beneficial manner, and the distributed part is very
helpful!
Richard J. Zak
-----Original Message-----
From: Darren Govoni [mailto:darren@ontrenet.com]
Sent: Wednesday, January 21, 2009 08:08
To: core-user@hadoop.apache.org
Subject: Suitable for Hadoop?
Hi,
I have a task to process large quantities of files by converting them into
other formats. Each file is processed as a whole and converted to a target
format. Since there are 100's of GB of data I thought it suitable for
Hadoop, but the problem is, I don't think the files can be broken apart and
processed. For example, how would mapreduce work to convert a Word Document
to PDF if the file is reduced to blocks? I'm not sure that's possible, or is
it?
thanks for any advice
Darren
RE: Suitable for Hadoop?
Posted by "Zak, Richard [USA]" <za...@bah.com>.
You can do that. I did a Map/Reduce job for about 6 GB of PDFs to
concatenate them, and the New York times used Hadoop to process a few TB of
PDFs.
What I would do is this:
- Use the iText library, a Java library for PDF manipulation (don't know
what you would use for reading Word docs)
- Don't use any Reducers
- Have the input be a text file with the directory(ies) to process, since
the mapper takes in file contents (and you don't want to read in one line of
binary)
- Have the map process all contents for that one given directory from the
input text file
- Break down the documents into more directories to go easier on the memory
- Use Amazon's EC2, and the scripts in <hadoop_dir>/src/contrib/ec2/bin/
(there is a script which passes environment variables to launched instances,
modify the script to allow Hadoop to use more memory by setting the
HADOOP_HEAPSIZE environment variable and having the variable properly
passed)
I realize this isn't the strong point of Map/Reduce or Hadoop, but it still
uses the HDFS in a beneficial manner, and the distributed part is very
helpful!
Richard J. Zak
-----Original Message-----
From: Darren Govoni [mailto:darren@ontrenet.com]
Sent: Wednesday, January 21, 2009 08:08
To: core-user@hadoop.apache.org
Subject: Suitable for Hadoop?
Hi,
I have a task to process large quantities of files by converting them into
other formats. Each file is processed as a whole and converted to a target
format. Since there are 100's of GB of data I thought it suitable for
Hadoop, but the problem is, I don't think the files can be broken apart and
processed. For example, how would mapreduce work to convert a Word Document
to PDF if the file is reduced to blocks? I'm not sure that's possible, or is
it?
thanks for any advice
Darren