You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Steve Lewis <lo...@gmail.com> on 2012/02/01 05:11:01 UTC
Any samples of how to write a custom FileSystem
Specifically how do I register a Custom FileSystem - any sample code
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Any samples of how to write a custom FileSystem
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Steven,
You could also look at HttpFSFilesystem in the hadoop-httpfs module, it is
quite simple and selfcontained.
Cheers.
Alejandro
On Tue, Jan 31, 2012 at 8:37 PM, Harsh J <ha...@cloudera.com> wrote:
> To write a custom filesystem, extend on the FileSystem class.
>
> Depending on the scheme it is supposed to serve, creating an entry
> "fs.<scheme>.impl" in core-site.xml, and then loading it via the
> FileSystem.get(URI, conf) API will auto load it for you, provided the
> URI you pass has the right scheme.
>
> So supposing I have a FS scheme "foo", I'd register it in core-site.xml as:
>
> <name>fs.foo.impl</name>
> <value>com.myorg.FooFileSystem</value>
>
> And then with a URI object from a path that goes "foo:///mypath/", I'd
> do: FileSystem.get(URI, new Configuration()) to get a FooFileSystem
> instance.
>
> Similarly, if you want to overload the local filesystem with your
> class, override the "fs.file.impl" config with your derivative class,
> and that'd be used in your configuration loaded programs in future.
>
> A good, not-so-complex impl. example to look at generally would be the
> S3FileSystem.
>
> On Wed, Feb 1, 2012 at 9:41 AM, Steve Lewis <lo...@gmail.com> wrote:
> > Specifically how do I register a Custom FileSystem - any sample code
> >
> > --
> > Steven M. Lewis PhD
> > 4221 105th Ave NE
> > Kirkland, WA 98033
> > 206-384-1340 (cell)
> > Skype lordjoe_com
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>
Re: Any samples of how to write a custom FileSystem
Posted by Harsh J <ha...@cloudera.com>.
To write a custom filesystem, extend on the FileSystem class.
Depending on the scheme it is supposed to serve, creating an entry
"fs.<scheme>.impl" in core-site.xml, and then loading it via the
FileSystem.get(URI, conf) API will auto load it for you, provided the
URI you pass has the right scheme.
So supposing I have a FS scheme "foo", I'd register it in core-site.xml as:
<name>fs.foo.impl</name>
<value>com.myorg.FooFileSystem</value>
And then with a URI object from a path that goes "foo:///mypath/", I'd
do: FileSystem.get(URI, new Configuration()) to get a FooFileSystem
instance.
Similarly, if you want to overload the local filesystem with your
class, override the "fs.file.impl" config with your derivative class,
and that'd be used in your configuration loaded programs in future.
A good, not-so-complex impl. example to look at generally would be the
S3FileSystem.
On Wed, Feb 1, 2012 at 9:41 AM, Steve Lewis <lo...@gmail.com> wrote:
> Specifically how do I register a Custom FileSystem - any sample code
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about