You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Samangooei Sina <ss...@ecs.soton.ac.uk> on 2010/10/02 14:21:16 UTC
Generating an Index for sequence files
Hello,
Using a map reduce and the image list from image-net I have created a set of sequence files which contain roughly 11 million images (split across roughly 100 part files). holding the images in sequence files allows for analysis using map reduce tasks, which is nice. However, I'd like other functionality. For example, the ability to pull out a given image by ID, for serving on a HTTP server . Also i'd like the ability to extract random subsets of the image set for analysis outside hadoop. The easiest way i've come up with to do this involves firstly knowing how many items there are exactly across my sequence files.
Both counting the images across my sequence files and extracting an individual image by ID has proved to be quite unsupported with sequence files alone. Therefore following the methods applies by the fellows over at cloud9 http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/ I've been trying to create an index of my sequence files. In a simple form this would be a text file, one line per ID, with the sequence file and offset of the given image. I've tried to implement a map-reduce task to generate this index
However, this is where i've come across some real problems. Firstly (short of some really horrible reflection trickery) it seems quite difficult to extract the offset into the sequence file of a given record during a map-reduce. This seemed to be supported in older versions of hadoop, and looking at the cloud9 sourcecode it would seem this functionality was how they generated their index, specifically using the RecordReader of the old style mapper interface. However, in the new version of hadoop (0.20, mapreduce package rather than mapred) I've been having some real problems extracting the information i need to create the index i'm trying to generate.
So to summaries:
- I want an index with which i can more quickly access sequence file entries
- I want an index to be able to quickly gain information such as "how many records are in these sequence files" so i can do funky things like "extract 10,000 random images from this set of 11 million"
How can i go about achieving these goals in the current version of hadoop
Thanks!
- Sina
Re: Generating an Index for sequence files
Posted by Harsh J <qw...@gmail.com>.
No it does not mean you can't use them in map reduce operations, and they
are especially built with that in mind.
InputFormat generally wraps over a simple Reader class (of the file format
it is of). Its not difficult to write one. Considering your specific
requirements of reading files, you may also find it better to write your own
input format classes for TFiles.
MapFiles are essentially SequenceFiles as already explained and can be used
with the same's IF class in map reduce operations. For fine tuned reading of
a map file, you will need your own impl, which isn't hard to do either.
Hadoop is very modular at the IO level.
Please look into the Reader and Writer class impls or API of each of the
file format you are interested in, then writing an input format class should
be doable enough.
On Oct 4, 2010 2:34 PM, "Sina Samangooei" <ss...@ecs.soton.ac.uk> wrote:
Hi,
Thanks for the Quick response.
It's good that there are provisions being made for the kind of problem i'm
trying to solve. However, I can't seem to find any sort of TFileInputFormat
or MapFileInputFormat. Does this mean TFiles and MapFiles can't be
simultaneously used for random access as well as map reduce tasks?
If this is the case TFiles and MapFiles are not suitable for my purposes. I
require the ability to perform large scale map-reduce operations on ALL of
the files, while at the same time having the ability to quickly access an
individual file. Two separate use cases, but both quite important. An option
might be to duplicate the data? Literally hold two copies, but that just
doesn't sit right.
Therefore, for now at least, i will continue with my index generation
scheme, i think i've found a work around that involves generating the index
outside of hadoop (i.e. not through a map-reduce task). This is slightly
slower than generating the index as part of a map reduce task, but once
generated the index should make access of files and various other operations
much faster
Thanks again,
- Sina
On 2 Oct 2010, at 17:36, Owen O'Malley wrote:
> On Sat, Oct 2, 2010 at 5:25 AM, Harsh J <qwertyman...
Re: Generating an Index for sequence files
Posted by Sina Samangooei <ss...@ecs.soton.ac.uk>.
Hi,
Thanks for the Quick response.
It's good that there are provisions being made for the kind of problem
i'm trying to solve. However, I can't seem to find any sort of
TFileInputFormat or MapFileInputFormat. Does this mean TFiles and
MapFiles can't be simultaneously used for random access as well as map
reduce tasks?
If this is the case TFiles and MapFiles are not suitable for my
purposes. I require the ability to perform large scale map-reduce
operations on ALL of the files, while at the same time having the
ability to quickly access an individual file. Two separate use cases,
but both quite important. An option might be to duplicate the data?
Literally hold two copies, but that just doesn't sit right.
Therefore, for now at least, i will continue with my index generation
scheme, i think i've found a work around that involves generating the
index outside of hadoop (i.e. not through a map-reduce task). This is
slightly slower than generating the index as part of a map reduce
task, but once generated the index should make access of files and
various other operations much faster
Thanks again,
- Sina
On 2 Oct 2010, at 17:36, Owen O'Malley wrote:
> On Sat, Oct 2, 2010 at 5:25 AM, Harsh J <qw...@gmail.com>
> wrote:
>> Maybe you should take a look at the TFile classes?
>
> The TFiles give you the meta information you want including row counts
> and an index that is integrated with the compression. The only
> downside is that you'll need to handle the serialization yourself,
> because TFiles only handle binary data. I'm working on a patch that
> include OFiles, which are TFiles that include serialization. The patch
> also includes support for Writables, Avro, ProtocolBuffers, and Thrift
> in SequenceFiles, MapFiles, and OFiles. (See
> https://issues.apache.org/jira/browse/HADOOP-6685 .)
>
> MapFiles are SequenceFiles with an index. (They are actually are
> implemented as two SequenceFiles, one as the index (key and position)
> and one as the data (key and value). MapFiles don't record the number
> of rows.
>
> -- Owen
Re: Generating an Index for sequence files
Posted by Owen O'Malley <om...@apache.org>.
On Sat, Oct 2, 2010 at 5:25 AM, Harsh J <qw...@gmail.com> wrote:
> Maybe you should take a look at the TFile classes?
The TFiles give you the meta information you want including row counts
and an index that is integrated with the compression. The only
downside is that you'll need to handle the serialization yourself,
because TFiles only handle binary data. I'm working on a patch that
include OFiles, which are TFiles that include serialization. The patch
also includes support for Writables, Avro, ProtocolBuffers, and Thrift
in SequenceFiles, MapFiles, and OFiles. (See
https://issues.apache.org/jira/browse/HADOOP-6685 .)
MapFiles are SequenceFiles with an index. (They are actually are
implemented as two SequenceFiles, one as the index (key and position)
and one as the data (key and value). MapFiles don't record the number
of rows.
-- Owen
Re: Generating an Index for sequence files
Posted by Harsh J <qw...@gmail.com>.
Maybe you should take a look at the TFile classes?
On Oct 2, 2010 5:52 PM, "Samangooei Sina" <ss...@ecs.soton.ac.uk> wrote:
Hello,
Using a map reduce and the image list from image-net I have created a set of
sequence files which contain roughly 11 million images (split across roughly
100 part files). holding the images in sequence files allows for analysis
using map reduce tasks, which is nice. However, I'd like other
functionality. For example, the ability to pull out a given image by ID, for
serving on a HTTP server . Also i'd like the ability to extract random
subsets of the image set for analysis outside hadoop. The easiest way i've
come up with to do this involves firstly knowing how many items there are
exactly across my sequence files.
Both counting the images across my sequence files and extracting an
individual image by ID has proved to be quite unsupported with sequence
files alone. Therefore following the methods applies by the fellows over at
cloud9 http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/ I've been trying to
create an index of my sequence files. In a simple form this would be a text
file, one line per ID, with the sequence file and offset of the given image.
I've tried to implement a map-reduce task to generate this index
However, this is where i've come across some real problems. Firstly (short
of some really horrible reflection trickery) it seems quite difficult to
extract the offset into the sequence file of a given record during a
map-reduce. This seemed to be supported in older versions of hadoop, and
looking at the cloud9 sourcecode it would seem this functionality was how
they generated their index, specifically using the RecordReader of the old
style mapper interface. However, in the new version of hadoop (0.20,
mapreduce package rather than mapred) I've been having some real problems
extracting the information i need to create the index i'm trying to
generate.
So to summaries:
- I want an index with which i can more quickly access sequence file entries
- I want an index to be able to quickly gain information such as "how many
records are in these sequence files" so i can do funky things like "extract
10,000 random images from this set of 11 million"
How can i go about achieving these goals in the current version of hadoop
Thanks!
- Sina