You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Vikrant Dhimate <vi...@gmail.com> on 2014/10/30 18:52:24 UTC
About building dummy Custom Distributed File system for hadoop.
We are working on B.Tech project and we want to build dummy Custom
Distributed File system for hadoop.
In 2.5.1 version there are 2 classes : 1) FileSystem and 2) AbstractFileSystem.
To make our own custom DFS which one of these two classes should we
implement or extend to our custom filesystem class. Because in earlier
version there was only FileSystem class.
Can you please tell us what is difference between FileSystem and
AbstractFileSystem classes and when to use each one sand which one should we
implement in our case?
Thanks,
Vikrant
Re: About building dummy Custom Distributed File system for hadoop.
Posted by Thomas Demoor <th...@amplidata.com>.
Hi Vikrant,
YARN uses AbstractFileSystem. Filesystem is the legacy interface (Hadoop
v1).
However, there is an abstract class DelegateToFileSystem that (trivially)
enables using a legacy FileSystem in YARN (see f.i. the Wasb class in
hadoop-azure)
So what you should implement depends on if you want to run YARN, legacy
Hadoop or both. Distributed filesystems are hard, nice to see you are
ambitious.
Good luck,
Thomas
On Thu, Oct 30, 2014 at 6:52 PM, Vikrant Dhimate <vi...@gmail.com>
wrote:
> We are working on B.Tech project and we want to build dummy Custom
> Distributed File system for hadoop.
>
> In 2.5.1 version there are 2 classes : 1) FileSystem and 2)
> AbstractFileSystem.
> To make our own custom DFS which one of these two classes should we
> implement or extend to our custom filesystem class. Because in earlier
> version there was only FileSystem class.
>
> Can you please tell us what is difference between FileSystem and
> AbstractFileSystem classes and when to use each one sand which one should
> we
> implement in our case?
>
> Thanks,
> Vikrant
>
>
Re: About building dummy Custom Distributed File system for hadoop.
Posted by Thomas Demoor <th...@amplidata.com>.
Hi Vikrant,
YARN uses AbstractFileSystem. Filesystem is the legacy interface (Hadoop
v1).
However, there is an abstract class DelegateToFileSystem that (trivially)
enables using a legacy FileSystem in YARN (see f.i. the Wasb class in
hadoop-azure)
So what you should implement depends on if you want to run YARN, legacy
Hadoop or both. Distributed filesystems are hard, nice to see you are
ambitious.
Good luck,
Thomas
On Thu, Oct 30, 2014 at 6:52 PM, Vikrant Dhimate <vi...@gmail.com>
wrote:
> We are working on B.Tech project and we want to build dummy Custom
> Distributed File system for hadoop.
>
> In 2.5.1 version there are 2 classes : 1) FileSystem and 2)
> AbstractFileSystem.
> To make our own custom DFS which one of these two classes should we
> implement or extend to our custom filesystem class. Because in earlier
> version there was only FileSystem class.
>
> Can you please tell us what is difference between FileSystem and
> AbstractFileSystem classes and when to use each one sand which one should
> we
> implement in our case?
>
> Thanks,
> Vikrant
>
>
Re: About building dummy Custom Distributed File system for hadoop.
Posted by Thomas Demoor <th...@amplidata.com>.
Hi Vikrant,
YARN uses AbstractFileSystem. Filesystem is the legacy interface (Hadoop
v1).
However, there is an abstract class DelegateToFileSystem that (trivially)
enables using a legacy FileSystem in YARN (see f.i. the Wasb class in
hadoop-azure)
So what you should implement depends on if you want to run YARN, legacy
Hadoop or both. Distributed filesystems are hard, nice to see you are
ambitious.
Good luck,
Thomas
On Thu, Oct 30, 2014 at 6:52 PM, Vikrant Dhimate <vi...@gmail.com>
wrote:
> We are working on B.Tech project and we want to build dummy Custom
> Distributed File system for hadoop.
>
> In 2.5.1 version there are 2 classes : 1) FileSystem and 2)
> AbstractFileSystem.
> To make our own custom DFS which one of these two classes should we
> implement or extend to our custom filesystem class. Because in earlier
> version there was only FileSystem class.
>
> Can you please tell us what is difference between FileSystem and
> AbstractFileSystem classes and when to use each one sand which one should
> we
> implement in our case?
>
> Thanks,
> Vikrant
>
>
Re: About building dummy Custom Distributed File system for hadoop.
Posted by Thomas Demoor <th...@amplidata.com>.
Hi Vikrant,
YARN uses AbstractFileSystem. Filesystem is the legacy interface (Hadoop
v1).
However, there is an abstract class DelegateToFileSystem that (trivially)
enables using a legacy FileSystem in YARN (see f.i. the Wasb class in
hadoop-azure)
So what you should implement depends on if you want to run YARN, legacy
Hadoop or both. Distributed filesystems are hard, nice to see you are
ambitious.
Good luck,
Thomas
On Thu, Oct 30, 2014 at 6:52 PM, Vikrant Dhimate <vi...@gmail.com>
wrote:
> We are working on B.Tech project and we want to build dummy Custom
> Distributed File system for hadoop.
>
> In 2.5.1 version there are 2 classes : 1) FileSystem and 2)
> AbstractFileSystem.
> To make our own custom DFS which one of these two classes should we
> implement or extend to our custom filesystem class. Because in earlier
> version there was only FileSystem class.
>
> Can you please tell us what is difference between FileSystem and
> AbstractFileSystem classes and when to use each one sand which one should
> we
> implement in our case?
>
> Thanks,
> Vikrant
>
>