You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Agarwal, Nikhil" <Ni...@netapp.com> on 2013/02/21 11:48:45 UTC
How to add another file system in Hadoop
Hi,
I am planning to add a file system called CDMI under org.apache.hadoop.fs in Hadoop, something similar to KFS or S3 which are already there under org.apache.hadoop.fs. I wanted to ask that say, I write my file system for CDMI and add the package under fs but then how do I tell the core-site.xml or other configuration files to use CDMI file system. Where all do I need to make changes to enable CDMI file system become a part of Hadoop ?
Thanks a lot in advance.
Regards,
Nikhil
Re: How to add another file system in Hadoop
Posted by Ling Kun <er...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 9:14 PM, Ling Kun <lk...@gmail.com> wrote:
> Hi Agarwal,
> This repository and the corresponding README file may give you some
> hint for the configuration.
>
> https://github.com/gluster/hadoop-glusterfs
>
> yours,
> Kun Ling
>
>
>
> On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
>> the classname of the custom FS impl.) is correct for the 1.x releases.
>> In 2.x and ahead, the class for a URI is auto-discovered from the
>> classpath (a 'service'). So as long as your jar is present on the
>> user's runtime, the FS class implementing cdmi:// will get loaded and
>> work without requiring a config injection globally like 1.x did.
>>
>> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>> > I may be guessing here a bit. Basically a filesystem is identified by
>> the
>> > protocol part of the URI of a file - so a file on the S3 filesystem will
>> > have a URI like s3://... If you look at the core-default.xml file in
>> Hadoop
>> > source, you will see configuration keys like fs.<fs-type>.impl and the
>> value
>> > is a class that implements that type of filesystem. So, you'd basically
>> have
>> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
>> > Hadoop configuration.
>> >
>> > Thanks
>> > hemanth
>> >
>> >
>> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
>> Nikhil.Agarwal@netapp.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >>
>> >>
>> >> I am planning to add a file system called CDMI under
>> org.apache.hadoop.fs
>> >> in Hadoop, something similar to KFS or S3 which are already there under
>> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
>> for
>> >> CDMI and add the package under fs but then how do I tell the
>> core-site.xml
>> >> or other configuration files to use CDMI file system. Where all do I
>> need to
>> >> make changes to enable CDMI file system become a part of Hadoop ?
>> >>
>> >>
>> >>
>> >> Thanks a lot in advance.
>> >>
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Nikhil
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> http://www.lingcc.com
>
> --
> http://www.lingcc.com <http://www.lingcc.com>
>
Re: How to add another file system in Hadoop
Posted by Ling Kun <er...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 9:14 PM, Ling Kun <lk...@gmail.com> wrote:
> Hi Agarwal,
> This repository and the corresponding README file may give you some
> hint for the configuration.
>
> https://github.com/gluster/hadoop-glusterfs
>
> yours,
> Kun Ling
>
>
>
> On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
>> the classname of the custom FS impl.) is correct for the 1.x releases.
>> In 2.x and ahead, the class for a URI is auto-discovered from the
>> classpath (a 'service'). So as long as your jar is present on the
>> user's runtime, the FS class implementing cdmi:// will get loaded and
>> work without requiring a config injection globally like 1.x did.
>>
>> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>> > I may be guessing here a bit. Basically a filesystem is identified by
>> the
>> > protocol part of the URI of a file - so a file on the S3 filesystem will
>> > have a URI like s3://... If you look at the core-default.xml file in
>> Hadoop
>> > source, you will see configuration keys like fs.<fs-type>.impl and the
>> value
>> > is a class that implements that type of filesystem. So, you'd basically
>> have
>> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
>> > Hadoop configuration.
>> >
>> > Thanks
>> > hemanth
>> >
>> >
>> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
>> Nikhil.Agarwal@netapp.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >>
>> >>
>> >> I am planning to add a file system called CDMI under
>> org.apache.hadoop.fs
>> >> in Hadoop, something similar to KFS or S3 which are already there under
>> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
>> for
>> >> CDMI and add the package under fs but then how do I tell the
>> core-site.xml
>> >> or other configuration files to use CDMI file system. Where all do I
>> need to
>> >> make changes to enable CDMI file system become a part of Hadoop ?
>> >>
>> >>
>> >>
>> >> Thanks a lot in advance.
>> >>
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Nikhil
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> http://www.lingcc.com
>
> --
> http://www.lingcc.com <http://www.lingcc.com>
>
Re: How to add another file system in Hadoop
Posted by Ling Kun <er...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 9:14 PM, Ling Kun <lk...@gmail.com> wrote:
> Hi Agarwal,
> This repository and the corresponding README file may give you some
> hint for the configuration.
>
> https://github.com/gluster/hadoop-glusterfs
>
> yours,
> Kun Ling
>
>
>
> On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
>> the classname of the custom FS impl.) is correct for the 1.x releases.
>> In 2.x and ahead, the class for a URI is auto-discovered from the
>> classpath (a 'service'). So as long as your jar is present on the
>> user's runtime, the FS class implementing cdmi:// will get loaded and
>> work without requiring a config injection globally like 1.x did.
>>
>> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>> > I may be guessing here a bit. Basically a filesystem is identified by
>> the
>> > protocol part of the URI of a file - so a file on the S3 filesystem will
>> > have a URI like s3://... If you look at the core-default.xml file in
>> Hadoop
>> > source, you will see configuration keys like fs.<fs-type>.impl and the
>> value
>> > is a class that implements that type of filesystem. So, you'd basically
>> have
>> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
>> > Hadoop configuration.
>> >
>> > Thanks
>> > hemanth
>> >
>> >
>> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
>> Nikhil.Agarwal@netapp.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >>
>> >>
>> >> I am planning to add a file system called CDMI under
>> org.apache.hadoop.fs
>> >> in Hadoop, something similar to KFS or S3 which are already there under
>> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
>> for
>> >> CDMI and add the package under fs but then how do I tell the
>> core-site.xml
>> >> or other configuration files to use CDMI file system. Where all do I
>> need to
>> >> make changes to enable CDMI file system become a part of Hadoop ?
>> >>
>> >>
>> >>
>> >> Thanks a lot in advance.
>> >>
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Nikhil
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> http://www.lingcc.com
>
> --
> http://www.lingcc.com <http://www.lingcc.com>
>
Re: How to add another file system in Hadoop
Posted by Ling Kun <er...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 9:14 PM, Ling Kun <lk...@gmail.com> wrote:
> Hi Agarwal,
> This repository and the corresponding README file may give you some
> hint for the configuration.
>
> https://github.com/gluster/hadoop-glusterfs
>
> yours,
> Kun Ling
>
>
>
> On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
>> the classname of the custom FS impl.) is correct for the 1.x releases.
>> In 2.x and ahead, the class for a URI is auto-discovered from the
>> classpath (a 'service'). So as long as your jar is present on the
>> user's runtime, the FS class implementing cdmi:// will get loaded and
>> work without requiring a config injection globally like 1.x did.
>>
>> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>> > I may be guessing here a bit. Basically a filesystem is identified by
>> the
>> > protocol part of the URI of a file - so a file on the S3 filesystem will
>> > have a URI like s3://... If you look at the core-default.xml file in
>> Hadoop
>> > source, you will see configuration keys like fs.<fs-type>.impl and the
>> value
>> > is a class that implements that type of filesystem. So, you'd basically
>> have
>> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
>> > Hadoop configuration.
>> >
>> > Thanks
>> > hemanth
>> >
>> >
>> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
>> Nikhil.Agarwal@netapp.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >>
>> >>
>> >> I am planning to add a file system called CDMI under
>> org.apache.hadoop.fs
>> >> in Hadoop, something similar to KFS or S3 which are already there under
>> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
>> for
>> >> CDMI and add the package under fs but then how do I tell the
>> core-site.xml
>> >> or other configuration files to use CDMI file system. Where all do I
>> need to
>> >> make changes to enable CDMI file system become a part of Hadoop ?
>> >>
>> >>
>> >>
>> >> Thanks a lot in advance.
>> >>
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Nikhil
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> http://www.lingcc.com
>
> --
> http://www.lingcc.com <http://www.lingcc.com>
>
Re: How to add another file system in Hadoop
Posted by Ling Kun <lk...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
> the classname of the custom FS impl.) is correct for the 1.x releases.
> In 2.x and ahead, the class for a URI is auto-discovered from the
> classpath (a 'service'). So as long as your jar is present on the
> user's runtime, the FS class implementing cdmi:// will get loaded and
> work without requiring a config injection globally like 1.x did.
>
> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > I may be guessing here a bit. Basically a filesystem is identified by the
> > protocol part of the URI of a file - so a file on the S3 filesystem will
> > have a URI like s3://... If you look at the core-default.xml file in
> Hadoop
> > source, you will see configuration keys like fs.<fs-type>.impl and the
> value
> > is a class that implements that type of filesystem. So, you'd basically
> have
> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> > Hadoop configuration.
> >
> > Thanks
> > hemanth
> >
> >
> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
> Nikhil.Agarwal@netapp.com>
> > wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> I am planning to add a file system called CDMI under
> org.apache.hadoop.fs
> >> in Hadoop, something similar to KFS or S3 which are already there under
> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
> for
> >> CDMI and add the package under fs but then how do I tell the
> core-site.xml
> >> or other configuration files to use CDMI file system. Where all do I
> need to
> >> make changes to enable CDMI file system become a part of Hadoop ?
> >>
> >>
> >>
> >> Thanks a lot in advance.
> >>
> >>
> >>
> >> Regards,
> >>
> >> Nikhil
> >
> >
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: How to add another file system in Hadoop
Posted by Ling Kun <lk...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
> the classname of the custom FS impl.) is correct for the 1.x releases.
> In 2.x and ahead, the class for a URI is auto-discovered from the
> classpath (a 'service'). So as long as your jar is present on the
> user's runtime, the FS class implementing cdmi:// will get loaded and
> work without requiring a config injection globally like 1.x did.
>
> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > I may be guessing here a bit. Basically a filesystem is identified by the
> > protocol part of the URI of a file - so a file on the S3 filesystem will
> > have a URI like s3://... If you look at the core-default.xml file in
> Hadoop
> > source, you will see configuration keys like fs.<fs-type>.impl and the
> value
> > is a class that implements that type of filesystem. So, you'd basically
> have
> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> > Hadoop configuration.
> >
> > Thanks
> > hemanth
> >
> >
> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
> Nikhil.Agarwal@netapp.com>
> > wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> I am planning to add a file system called CDMI under
> org.apache.hadoop.fs
> >> in Hadoop, something similar to KFS or S3 which are already there under
> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
> for
> >> CDMI and add the package under fs but then how do I tell the
> core-site.xml
> >> or other configuration files to use CDMI file system. Where all do I
> need to
> >> make changes to enable CDMI file system become a part of Hadoop ?
> >>
> >>
> >>
> >> Thanks a lot in advance.
> >>
> >>
> >>
> >> Regards,
> >>
> >> Nikhil
> >
> >
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: How to add another file system in Hadoop
Posted by Ling Kun <lk...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
> the classname of the custom FS impl.) is correct for the 1.x releases.
> In 2.x and ahead, the class for a URI is auto-discovered from the
> classpath (a 'service'). So as long as your jar is present on the
> user's runtime, the FS class implementing cdmi:// will get loaded and
> work without requiring a config injection globally like 1.x did.
>
> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > I may be guessing here a bit. Basically a filesystem is identified by the
> > protocol part of the URI of a file - so a file on the S3 filesystem will
> > have a URI like s3://... If you look at the core-default.xml file in
> Hadoop
> > source, you will see configuration keys like fs.<fs-type>.impl and the
> value
> > is a class that implements that type of filesystem. So, you'd basically
> have
> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> > Hadoop configuration.
> >
> > Thanks
> > hemanth
> >
> >
> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
> Nikhil.Agarwal@netapp.com>
> > wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> I am planning to add a file system called CDMI under
> org.apache.hadoop.fs
> >> in Hadoop, something similar to KFS or S3 which are already there under
> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
> for
> >> CDMI and add the package under fs but then how do I tell the
> core-site.xml
> >> or other configuration files to use CDMI file system. Where all do I
> need to
> >> make changes to enable CDMI file system become a part of Hadoop ?
> >>
> >>
> >>
> >> Thanks a lot in advance.
> >>
> >>
> >>
> >> Regards,
> >>
> >> Nikhil
> >
> >
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: How to add another file system in Hadoop
Posted by Ling Kun <lk...@gmail.com>.
Hi Agarwal,
This repository and the corresponding README file may give you some hint
for the configuration.
https://github.com/gluster/hadoop-glusterfs
yours,
Kun Ling
On Thu, Feb 21, 2013 at 8:53 PM, Harsh J <ha...@cloudera.com> wrote:
> What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
> the classname of the custom FS impl.) is correct for the 1.x releases.
> In 2.x and ahead, the class for a URI is auto-discovered from the
> classpath (a 'service'). So as long as your jar is present on the
> user's runtime, the FS class implementing cdmi:// will get loaded and
> work without requiring a config injection globally like 1.x did.
>
> On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > I may be guessing here a bit. Basically a filesystem is identified by the
> > protocol part of the URI of a file - so a file on the S3 filesystem will
> > have a URI like s3://... If you look at the core-default.xml file in
> Hadoop
> > source, you will see configuration keys like fs.<fs-type>.impl and the
> value
> > is a class that implements that type of filesystem. So, you'd basically
> have
> > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> > Hadoop configuration.
> >
> > Thanks
> > hemanth
> >
> >
> > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <
> Nikhil.Agarwal@netapp.com>
> > wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> I am planning to add a file system called CDMI under
> org.apache.hadoop.fs
> >> in Hadoop, something similar to KFS or S3 which are already there under
> >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system
> for
> >> CDMI and add the package under fs but then how do I tell the
> core-site.xml
> >> or other configuration files to use CDMI file system. Where all do I
> need to
> >> make changes to enable CDMI file system become a part of Hadoop ?
> >>
> >>
> >>
> >> Thanks a lot in advance.
> >>
> >>
> >>
> >> Regards,
> >>
> >> Nikhil
> >
> >
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: How to add another file system in Hadoop
Posted by Harsh J <ha...@cloudera.com>.
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
the classname of the custom FS impl.) is correct for the 1.x releases.
In 2.x and ahead, the class for a URI is auto-discovered from the
classpath (a 'service'). So as long as your jar is present on the
user's runtime, the FS class implementing cdmi:// will get loaded and
work without requiring a config injection globally like 1.x did.
On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> I may be guessing here a bit. Basically a filesystem is identified by the
> protocol part of the URI of a file - so a file on the S3 filesystem will
> have a URI like s3://... If you look at the core-default.xml file in Hadoop
> source, you will see configuration keys like fs.<fs-type>.impl and the value
> is a class that implements that type of filesystem. So, you'd basically have
> to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> Hadoop configuration.
>
> Thanks
> hemanth
>
>
> On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <Ni...@netapp.com>
> wrote:
>>
>> Hi,
>>
>>
>>
>> I am planning to add a file system called CDMI under org.apache.hadoop.fs
>> in Hadoop, something similar to KFS or S3 which are already there under
>> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
>> CDMI and add the package under fs but then how do I tell the core-site.xml
>> or other configuration files to use CDMI file system. Where all do I need to
>> make changes to enable CDMI file system become a part of Hadoop ?
>>
>>
>>
>> Thanks a lot in advance.
>>
>>
>>
>> Regards,
>>
>> Nikhil
>
>
--
Harsh J
Re: How to add another file system in Hadoop
Posted by Harsh J <ha...@cloudera.com>.
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
the classname of the custom FS impl.) is correct for the 1.x releases.
In 2.x and ahead, the class for a URI is auto-discovered from the
classpath (a 'service'). So as long as your jar is present on the
user's runtime, the FS class implementing cdmi:// will get loaded and
work without requiring a config injection globally like 1.x did.
On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> I may be guessing here a bit. Basically a filesystem is identified by the
> protocol part of the URI of a file - so a file on the S3 filesystem will
> have a URI like s3://... If you look at the core-default.xml file in Hadoop
> source, you will see configuration keys like fs.<fs-type>.impl and the value
> is a class that implements that type of filesystem. So, you'd basically have
> to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> Hadoop configuration.
>
> Thanks
> hemanth
>
>
> On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <Ni...@netapp.com>
> wrote:
>>
>> Hi,
>>
>>
>>
>> I am planning to add a file system called CDMI under org.apache.hadoop.fs
>> in Hadoop, something similar to KFS or S3 which are already there under
>> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
>> CDMI and add the package under fs but then how do I tell the core-site.xml
>> or other configuration files to use CDMI file system. Where all do I need to
>> make changes to enable CDMI file system become a part of Hadoop ?
>>
>>
>>
>> Thanks a lot in advance.
>>
>>
>>
>> Regards,
>>
>> Nikhil
>
>
--
Harsh J
Re: How to add another file system in Hadoop
Posted by Harsh J <ha...@cloudera.com>.
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
the classname of the custom FS impl.) is correct for the 1.x releases.
In 2.x and ahead, the class for a URI is auto-discovered from the
classpath (a 'service'). So as long as your jar is present on the
user's runtime, the FS class implementing cdmi:// will get loaded and
work without requiring a config injection globally like 1.x did.
On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> I may be guessing here a bit. Basically a filesystem is identified by the
> protocol part of the URI of a file - so a file on the S3 filesystem will
> have a URI like s3://... If you look at the core-default.xml file in Hadoop
> source, you will see configuration keys like fs.<fs-type>.impl and the value
> is a class that implements that type of filesystem. So, you'd basically have
> to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> Hadoop configuration.
>
> Thanks
> hemanth
>
>
> On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <Ni...@netapp.com>
> wrote:
>>
>> Hi,
>>
>>
>>
>> I am planning to add a file system called CDMI under org.apache.hadoop.fs
>> in Hadoop, something similar to KFS or S3 which are already there under
>> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
>> CDMI and add the package under fs but then how do I tell the core-site.xml
>> or other configuration files to use CDMI file system. Where all do I need to
>> make changes to enable CDMI file system become a part of Hadoop ?
>>
>>
>>
>> Thanks a lot in advance.
>>
>>
>>
>> Regards,
>>
>> Nikhil
>
>
--
Harsh J
Re: How to add another file system in Hadoop
Posted by Harsh J <ha...@cloudera.com>.
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
the classname of the custom FS impl.) is correct for the 1.x releases.
In 2.x and ahead, the class for a URI is auto-discovered from the
classpath (a 'service'). So as long as your jar is present on the
user's runtime, the FS class implementing cdmi:// will get loaded and
work without requiring a config injection globally like 1.x did.
On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> I may be guessing here a bit. Basically a filesystem is identified by the
> protocol part of the URI of a file - so a file on the S3 filesystem will
> have a URI like s3://... If you look at the core-default.xml file in Hadoop
> source, you will see configuration keys like fs.<fs-type>.impl and the value
> is a class that implements that type of filesystem. So, you'd basically have
> to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> Hadoop configuration.
>
> Thanks
> hemanth
>
>
> On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <Ni...@netapp.com>
> wrote:
>>
>> Hi,
>>
>>
>>
>> I am planning to add a file system called CDMI under org.apache.hadoop.fs
>> in Hadoop, something similar to KFS or S3 which are already there under
>> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
>> CDMI and add the package under fs but then how do I tell the core-site.xml
>> or other configuration files to use CDMI file system. Where all do I need to
>> make changes to enable CDMI file system become a part of Hadoop ?
>>
>>
>>
>> Thanks a lot in advance.
>>
>>
>>
>> Regards,
>>
>> Nikhil
>
>
--
Harsh J
Re: How to add another file system in Hadoop
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
I may be guessing here a bit. Basically a filesystem is identified by the
protocol part of the URI of a file - so a file on the S3 filesystem will
have a URI like s3://... If you look at the core-default.xml file in Hadoop
source, you will see configuration keys like fs.<fs-type>.impl and the
value is a class that implements that type of filesystem. So, you'd
basically have to define a cdmi:// type protocol and define a fs.cdmi.impl
key in your Hadoop configuration.
Thanks
hemanth
On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil
<Ni...@netapp.com>wrote:
> Hi,****
>
> ** **
>
> I am planning to add a file system called CDMI under org.apache.hadoop.fs
> in Hadoop, something similar to KFS or S3 which are already there under
> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
> CDMI and add the package under fs but then how do I tell the core-site.xml
> or other configuration files to use CDMI file system. Where all do I need
> to make changes to enable CDMI file system become a part of Hadoop ?****
>
> ** **
>
> Thanks a lot in advance.****
>
> ** **
>
> Regards,****
>
> Nikhil ****
>
Re: How to add another file system in Hadoop
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
I may be guessing here a bit. Basically a filesystem is identified by the
protocol part of the URI of a file - so a file on the S3 filesystem will
have a URI like s3://... If you look at the core-default.xml file in Hadoop
source, you will see configuration keys like fs.<fs-type>.impl and the
value is a class that implements that type of filesystem. So, you'd
basically have to define a cdmi:// type protocol and define a fs.cdmi.impl
key in your Hadoop configuration.
Thanks
hemanth
On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil
<Ni...@netapp.com>wrote:
> Hi,****
>
> ** **
>
> I am planning to add a file system called CDMI under org.apache.hadoop.fs
> in Hadoop, something similar to KFS or S3 which are already there under
> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
> CDMI and add the package under fs but then how do I tell the core-site.xml
> or other configuration files to use CDMI file system. Where all do I need
> to make changes to enable CDMI file system become a part of Hadoop ?****
>
> ** **
>
> Thanks a lot in advance.****
>
> ** **
>
> Regards,****
>
> Nikhil ****
>
Re: How to add another file system in Hadoop
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
I may be guessing here a bit. Basically a filesystem is identified by the
protocol part of the URI of a file - so a file on the S3 filesystem will
have a URI like s3://... If you look at the core-default.xml file in Hadoop
source, you will see configuration keys like fs.<fs-type>.impl and the
value is a class that implements that type of filesystem. So, you'd
basically have to define a cdmi:// type protocol and define a fs.cdmi.impl
key in your Hadoop configuration.
Thanks
hemanth
On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil
<Ni...@netapp.com>wrote:
> Hi,****
>
> ** **
>
> I am planning to add a file system called CDMI under org.apache.hadoop.fs
> in Hadoop, something similar to KFS or S3 which are already there under
> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
> CDMI and add the package under fs but then how do I tell the core-site.xml
> or other configuration files to use CDMI file system. Where all do I need
> to make changes to enable CDMI file system become a part of Hadoop ?****
>
> ** **
>
> Thanks a lot in advance.****
>
> ** **
>
> Regards,****
>
> Nikhil ****
>
Re: How to add another file system in Hadoop
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
I may be guessing here a bit. Basically a filesystem is identified by the
protocol part of the URI of a file - so a file on the S3 filesystem will
have a URI like s3://... If you look at the core-default.xml file in Hadoop
source, you will see configuration keys like fs.<fs-type>.impl and the
value is a class that implements that type of filesystem. So, you'd
basically have to define a cdmi:// type protocol and define a fs.cdmi.impl
key in your Hadoop configuration.
Thanks
hemanth
On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil
<Ni...@netapp.com>wrote:
> Hi,****
>
> ** **
>
> I am planning to add a file system called CDMI under org.apache.hadoop.fs
> in Hadoop, something similar to KFS or S3 which are already there under
> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
> CDMI and add the package under fs but then how do I tell the core-site.xml
> or other configuration files to use CDMI file system. Where all do I need
> to make changes to enable CDMI file system become a part of Hadoop ?****
>
> ** **
>
> Thanks a lot in advance.****
>
> ** **
>
> Regards,****
>
> Nikhil ****
>