You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Lara Dong <la...@gmail.com> on 2012/06/27 03:18:30 UTC
Sequence File Index
As now we use the Sequence File to solve the small file problems, but the
Sequence File has no index. What I mean is if I transfer my small file into
a Sequence File, how do I know where is the position to get my file in the
Sequence File? I know the Map File can avoid this problem, but I prefer to
use Sequence File. Just now I use Lucene to build an index to record the
position. Is there a better way?
Re: Sequence File Index
Posted by Lara Dong <la...@gmail.com>.
Yes, it is a solution. I just wonder if we use SequenceFile, how
to retrieve the original small files? except traversaling the
SequenceFile .
2012/6/27 Harsh J <ha...@cloudera.com>
> MapFiles were designed for this. It does something similar to what you
> have done, keeps an external index of keys. Why do you not wish to use
> MapFiles though?
>
> On Wed, Jun 27, 2012 at 6:48 AM, Lara Dong <la...@gmail.com> wrote:
> > As now we use the Sequence File to solve the small file problems, but the
> > Sequence File has no index. What I mean is if I transfer my small file
> into
> > a Sequence File, how do I know where is the position to get my file in
> the
> > Sequence File? I know the Map File can avoid this problem, but I prefer
> to
> > use Sequence File. Just now I use Lucene to build an index to record the
> > position. Is there a better way?
> >
>
>
>
> --
> Harsh J
>
Re: Sequence File Index
Posted by Harsh J <ha...@cloudera.com>.
MapFiles were designed for this. It does something similar to what you
have done, keeps an external index of keys. Why do you not wish to use
MapFiles though?
On Wed, Jun 27, 2012 at 6:48 AM, Lara Dong <la...@gmail.com> wrote:
> As now we use the Sequence File to solve the small file problems, but the
> Sequence File has no index. What I mean is if I transfer my small file into
> a Sequence File, how do I know where is the position to get my file in the
> Sequence File? I know the Map File can avoid this problem, but I prefer to
> use Sequence File. Just now I use Lucene to build an index to record the
> position. Is there a better way?
>
--
Harsh J