You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Jonathan Seidman <jo...@opendatagroup.com> on 2009/08/10 22:28:50 UTC
Contributing a New FileSystem Implementation - Questions About
Process
We're getting ready to contribute our FileSystem implementation for the
Sector DFS (sector.sf.net). Up to now our development and testing has been
against 0.18.3, so our intention was to first integrate with that release
and then work on integrating with the trunk for the next release. A couple
of questions:
* Should we integrate with the 0.18 branch, or just put our changes into the
0.18.3 release? We're not sure if there are plans for further releases on
the 0.18 branch.
* Should we create separate Jira's for the two releases, or can we create a
single Jira to cover the 0.18 and trunk releases?
Thanks for your help. We'd like to make sure we're facilitating the process,
so any suggestions are appreciated.
--
Jonathan Seidman
Open Data Group
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Jonathan Seidman <jo...@opendatagroup.com>.
Thanks for the pointers Tom and Cos, this is helpful. We'll take a look at
these as we firm up our test plan.
On Wed, Aug 12, 2009 at 6:58 AM, Tom White <to...@cloudera.com> wrote:
> Hi Jonathan,
>
> For testing you can subclass FileSystemContractBaseTest and create an
> instance of the filesystem to test in the set up method, to check that
> the new implementation conforms to the contract of the FileSystem
> interface (although see also
> https://issues.apache.org/jira/browse/HDFS-303).
>
> See TestHDFSFileSystemContract for an example.
>
> Cheers,
> Tom
>
> On Tue, Aug 11, 2009 at 5:13 PM, Konstantin Boudnik<co...@yahoo-inc.com>
> wrote:
> > Jonathan,
> >
> > in case you need to take a look at a common testplan template you can
> find
> > one in HDFS-265 or more generic in HADOOP-5587
> >
> > Cos
> >
> > On 8/11/09 8:32 AM, Steve Loughran wrote:
> >>
> >> Jonathan Seidman wrote:
> >>>
> >>> Thanks for the replies. We'll create a patch for trunk and then include
> a
> >>> 0.18 compatible patch with the Jira, as you suggest.
> >>>
> >>> Based on other contributed FileSystem implementations, we were assuming
> >>> this
> >>> should go in o.a.h.fs and not contrib, so thanks for the
> clarification.
> >>>
> >>
> >> what's the test plan going to be for a new FS?
> >
>
--
Jonathan Seidman
Open Data Group
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Tom White <to...@cloudera.com>.
Hi Jonathan,
For testing you can subclass FileSystemContractBaseTest and create an
instance of the filesystem to test in the set up method, to check that
the new implementation conforms to the contract of the FileSystem
interface (although see also
https://issues.apache.org/jira/browse/HDFS-303).
See TestHDFSFileSystemContract for an example.
Cheers,
Tom
On Tue, Aug 11, 2009 at 5:13 PM, Konstantin Boudnik<co...@yahoo-inc.com> wrote:
> Jonathan,
>
> in case you need to take a look at a common testplan template you can find
> one in HDFS-265 or more generic in HADOOP-5587
>
> Cos
>
> On 8/11/09 8:32 AM, Steve Loughran wrote:
>>
>> Jonathan Seidman wrote:
>>>
>>> Thanks for the replies. We'll create a patch for trunk and then include a
>>> 0.18 compatible patch with the Jira, as you suggest.
>>>
>>> Based on other contributed FileSystem implementations, we were assuming
>>> this
>>> should go in o.a.h.fs and not contrib, so thanks for the clarification.
>>>
>>
>> what's the test plan going to be for a new FS?
>
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Konstantin Boudnik <co...@yahoo-inc.com>.
Jonathan,
in case you need to take a look at a common testplan template you can find one
in HDFS-265 or more generic in HADOOP-5587
Cos
On 8/11/09 8:32 AM, Steve Loughran wrote:
> Jonathan Seidman wrote:
>> Thanks for the replies. We'll create a patch for trunk and then include a
>> 0.18 compatible patch with the Jira, as you suggest.
>>
>> Based on other contributed FileSystem implementations, we were assuming this
>> should go in o.a.h.fs and not contrib, so thanks for the clarification.
>>
>
> what's the test plan going to be for a new FS?
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Steve Loughran <st...@apache.org>.
Jonathan Seidman wrote:
> Thanks for the replies. We'll create a patch for trunk and then include a
> 0.18 compatible patch with the Jira, as you suggest.
>
> Based on other contributed FileSystem implementations, we were assuming this
> should go in o.a.h.fs and not contrib, so thanks for the clarification.
>
what's the test plan going to be for a new FS?
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Jonathan Seidman <jo...@opendatagroup.com>.
Thanks for the replies. We'll create a patch for trunk and then include a
0.18 compatible patch with the Jira, as you suggest.
Based on other contributed FileSystem implementations, we were assuming this
should go in o.a.h.fs and not contrib, so thanks for the clarification.
Jonathan
On Mon, Aug 10, 2009 at 6:37 PM, Chris Douglas <ch...@gmail.com>wrote:
> >> * Should we integrate with the 0.18 branch, or just put our changes into
> >> the
> >> 0.18.3 release? We're not sure if there are plans for further releases
> on
> >> the 0.18 branch.
>
> This will not be committed to the 0.18 branch, even if there is an
> 0.18.4 release. If you wanted to post an 0.18 compatible patch on the
> same JIRA for reference, that's fine, but only patches to trunk will
> be considered for inclusion.
>
> > In general you should be aware that you should integrate as a new project
> in
> > the contrib/ dir. If all goes well you shouldn't need to make any patches
> > that touch Hadoop itself.
>
> If this is a shim, it should probably live in o.a.h.fs with
> S3FileSystem, KosmosFileSystem, etc. rather than being added to
> contrib. -C
>
--
Jonathan Seidman
Open Data Group
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Chris Douglas <ch...@gmail.com>.
>> * Should we integrate with the 0.18 branch, or just put our changes into
>> the
>> 0.18.3 release? We're not sure if there are plans for further releases on
>> the 0.18 branch.
This will not be committed to the 0.18 branch, even if there is an
0.18.4 release. If you wanted to post an 0.18 compatible patch on the
same JIRA for reference, that's fine, but only patches to trunk will
be considered for inclusion.
> In general you should be aware that you should integrate as a new project in
> the contrib/ dir. If all goes well you shouldn't need to make any patches
> that touch Hadoop itself.
If this is a shim, it should probably live in o.a.h.fs with
S3FileSystem, KosmosFileSystem, etc. rather than being added to
contrib. -C
Re: Contributing a New FileSystem Implementation - Questions About
Process
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jonathan,
Responses inline below:
On Mon, Aug 10, 2009 at 1:28 PM, Jonathan Seidman <
jonathan.seidman@opendatagroup.com> wrote:
> We're getting ready to contribute our FileSystem implementation for the
> Sector DFS (sector.sf.net). Up to now our development and testing has been
> against 0.18.3, so our intention was to first integrate with that release
> and then work on integrating with the trunk for the next release. A couple
> of questions:
> * Should we integrate with the 0.18 branch, or just put our changes into
> the
> 0.18.3 release? We're not sure if there are plans for further releases on
> the 0.18 branch.
>
I believe there are plans for an 0.18.4 release eventually, though a new
feature like a new DFS is unlikely to be included. I would concentrate on
integrating against trunk rather than a past release. If trunk is
inconvenient since it's a moving target, you could integrate against
branch-20. While a lot of people are currently using 0.18, I anticipate that
most users will be switching over to 0.20 in the next couple of months.
>
> * Should we create separate Jira's for the two releases, or can we create a
> single Jira to cover the 0.18 and trunk releases?
>
Single JIRA should be fine - just post multiple patches.
> Thanks for your help. We'd like to make sure we're facilitating the
> process,
> so any suggestions are appreciated.
>
In general you should be aware that you should integrate as a new project in
the contrib/ dir. If all goes well you shouldn't need to make any patches
that touch Hadoop itself.
-Todd