You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Oleg Ruchovets <or...@gmail.com> on 2011/03/14 09:26:30 UTC
data and metadata in a different folders
Hi ,
Is it possible to configure hadoop (hdfs) in a such way that meta data
and data itself will be in a different folders?
Actually the questions came from Idea that: it is possible to use 2 file
system : one for data and one for metadata. We want to get an effect that
access to meta data is much more faster then to the data itself. Has someone
already implemented it or has such experience please advice.
Thanks in advance
Oleg.
Re: data and metadata in a different folders
Posted by Oleg Ruchovets <or...@gmail.com>.
basically what we want is to split the data stored by hadoop on regular
disks and put data used for searches on faster ssd disk.
if those two types of data can be separated then how to define it? if not,
is there anything used by hadoop/hdfs that would benefit from being on a
very fast disk (like cache for example) and is small enough - up to 80gb per
server.
Thanks.
Oleg.
2011/3/14 Ahmed Said Nagy <ah...@hotmail.com>
>
> It holds the directory structure and the file information like permissions
> and location of the file. This is usually held at the name node if u want to
> test that try to move a file that has non zero bytes and see what happens if
> it is not creates then propably it means that the master is unable to write
> on the node check the permissions I faced something very similar.Regards
>
>
>
>
>
>
> > From: qwertymaniac@gmail.com
> > Date: Mon, 14 Mar 2011 15:58:19 +0530
> > Subject: Re: data and metadata in a different folders
> > To: common-user@hadoop.apache.org
> >
> > What type of metadata? Stuff like what's held by sequence files? Is it
> > to be held for each file or per directory/set-of-files?
> >
> > On Mon, Mar 14, 2011 at 1:56 PM, Oleg Ruchovets <or...@gmail.com>
> wrote:
> > > Hi ,
> > > Is it possible to configure hadoop (hdfs) in a such way that meta
> data
> > > and data itself will be in a different folders?
> > >
> > > Actually the questions came from Idea that: it is possible to use 2
> file
> > > system : one for data and one for metadata. We want to get an effect
> that
> > > access to meta data is much more faster then to the data itself. Has
> someone
> > > already implemented it or has such experience please advice.
> > >
> > > Thanks in advance
> > > Oleg.
> > >
> >
> >
> >
> > --
> > Harsh J
> > http://harshj.com
>
>
RE: data and metadata in a different folders
Posted by Ahmed Said Nagy <ah...@hotmail.com>.
It holds the directory structure and the file information like permissions and location of the file. This is usually held at the name node if u want to test that try to move a file that has non zero bytes and see what happens if it is not creates then propably it means that the master is unable to write on the node check the permissions I faced something very similar.Regards
> From: qwertymaniac@gmail.com
> Date: Mon, 14 Mar 2011 15:58:19 +0530
> Subject: Re: data and metadata in a different folders
> To: common-user@hadoop.apache.org
>
> What type of metadata? Stuff like what's held by sequence files? Is it
> to be held for each file or per directory/set-of-files?
>
> On Mon, Mar 14, 2011 at 1:56 PM, Oleg Ruchovets <or...@gmail.com> wrote:
> > Hi ,
> > Is it possible to configure hadoop (hdfs) in a such way that meta data
> > and data itself will be in a different folders?
> >
> > Actually the questions came from Idea that: it is possible to use 2 file
> > system : one for data and one for metadata. We want to get an effect that
> > access to meta data is much more faster then to the data itself. Has someone
> > already implemented it or has such experience please advice.
> >
> > Thanks in advance
> > Oleg.
> >
>
>
>
> --
> Harsh J
> http://harshj.com
Re: data and metadata in a different folders
Posted by Harsh J <qw...@gmail.com>.
What type of metadata? Stuff like what's held by sequence files? Is it
to be held for each file or per directory/set-of-files?
On Mon, Mar 14, 2011 at 1:56 PM, Oleg Ruchovets <or...@gmail.com> wrote:
> Hi ,
> Is it possible to configure hadoop (hdfs) in a such way that meta data
> and data itself will be in a different folders?
>
> Actually the questions came from Idea that: it is possible to use 2 file
> system : one for data and one for metadata. We want to get an effect that
> access to meta data is much more faster then to the data itself. Has someone
> already implemented it or has such experience please advice.
>
> Thanks in advance
> Oleg.
>
--
Harsh J
http://harshj.com