You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/05/31 03:07:06 UTC
Backwards compatibility of FileSystem interface.
Hi :
Are FileSystem interfaces gauranteed to retain semantics of the old
FileSystem contract?
Im noticing that the new FileSystem uses the "CreateFlag" class, which
older ones did not, and I'm wondering wether the deprecation of some
FileSystem methods (i.e. createNonRecursive) will eventually be entirely
phased out.
This is in reference to some issues with HBASE and the createNonRecursive
implementation which varies across different hadoop versions.
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Backwards compatibility of FileSystem interface.
Posted by Steve Loughran <st...@hortonworks.com>.
(forwarding to common-dev which is where this discussion about APIs should
go)
1. FileSystem API may add new methods from version to version. There's
some list operation that's gone in too.
2. I don't think there are any guarantees that we don't add new methods
to a base class -it's why they are often preferred to an interface.
3. You are free to add the new methods into your implementation -just
don't mark them as @Override and they will compile, and should be picked up
in the later version
This adds an extra point to the compatibility JIRA: plugins.
"classes that subclass a Hadoop class to provide a plugin point should
expect to need recompiling on each major version, possibly with the
handling of changes to methods. There is no guarantee that a plugin built
for an earlier version of Hadoop can be directly used in the later version,
and implementors of plugins should expect to have to release new versions"
On 31 May 2013 14:58, Jay Vyas <ja...@gmail.com> wrote:
> For context - It appears that over time the FileSystem has changed the
> signature to the "createNonRecursive" method, (in 2.x there are 3
> signatures, in 1.x, there are 2 signatures).
>
> [root@jvyas hadoop-common]# git checkout branch-1.1
> Switched to branch 'branch-1.1'
> [root@jvyas hadoop-common]# grep NonRec
> ./src/core/org/apache/hadoop/fs/FileSystem.java
> public FSDataOutputStream createNonRecursive(Path f,
> return this.createNonRecursive(f, FsPermission.getDefault(),
> public FSDataOutputStream createNonRecursive(Path f, FsPermission
> permission,
> throw new IOException("createNonRecursive unsupported for this
> filesystem "
>
> [root@jvyas hadoop-common]# git checkout branch-2.1.0-alpha
> Switched to branch 'branch-2.1.0-alpha'
>
> [root@jvyas hadoop-common]# cat
>
> ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
> | grep NonRec
> | grep NonRec
>
> public FSDataOutputStream createNonRecursive(Path f,
> return this.createNonRecursive(f, FsPermission.getDefault(),
> public FSDataOutputStream createNonRecursive(Path f, FsPermission
> permission,
> return createNonRecursive(f, permission,
> public FSDataOutputStream createNonRecursive(Path f, FsPermission
> permission,
> throw new IOException("createNonRecursive unsupported for this
> filesystem
>
>
> So, this (im assuming) is to retain backwards compatibility. But it when
> new API calls are made older FileSystem implementations dont work anymore.
>
> For context, here is a closely related JIRA
>
>
> https://issues.apache.org/jira/browse/HADOOP-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>
>
> On Fri, May 31, 2013 at 6:09 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > Jay, this is is more a common-dev question, as it goes beyond hdfs
> >
> > On 31 May 2013 02:07, Jay Vyas <ja...@gmail.com> wrote:
> >
> > > Hi :
> > >
> > > Are FileSystem interfaces gauranteed to retain semantics of the old
> > > FileSystem contract?
> > >
> >
> > retain the explicit semantics, yes. Implementations may change some
> > implementation details which may becomes visible
> >
> >
> > >
> > > Im noticing that the new FileSystem uses the "CreateFlag" class, which
> > > older ones did not, and I'm wondering wether the deprecation of some
> > > FileSystem methods (i.e. createNonRecursive) will eventually be
> entirely
> > > phased out.
> > >
> > > This is in reference to some issues with HBASE and the
> createNonRecursive
> > > implementation which varies across different hadoop versions.
> > >
> > >
> > can you point to the JIRAs? I presume they are append related?
> >
> >
> > > --
> > > Jay Vyas
> > > http://jayunit100.blogspot.com
> > >
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Backwards compatibility of FileSystem interface.
Posted by Steve Loughran <st...@hortonworks.com>.
(forwarding to common-dev which is where this discussion about APIs should
go)
1. FileSystem API may add new methods from version to version. There's
some list operation that's gone in too.
2. I don't think there are any guarantees that we don't add new methods
to a base class -it's why they are often preferred to an interface.
3. You are free to add the new methods into your implementation -just
don't mark them as @Override and they will compile, and should be picked up
in the later version
This adds an extra point to the compatibility JIRA: plugins.
"classes that subclass a Hadoop class to provide a plugin point should
expect to need recompiling on each major version, possibly with the
handling of changes to methods. There is no guarantee that a plugin built
for an earlier version of Hadoop can be directly used in the later version,
and implementors of plugins should expect to have to release new versions"
On 31 May 2013 14:58, Jay Vyas <ja...@gmail.com> wrote:
> For context - It appears that over time the FileSystem has changed the
> signature to the "createNonRecursive" method, (in 2.x there are 3
> signatures, in 1.x, there are 2 signatures).
>
> [root@jvyas hadoop-common]# git checkout branch-1.1
> Switched to branch 'branch-1.1'
> [root@jvyas hadoop-common]# grep NonRec
> ./src/core/org/apache/hadoop/fs/FileSystem.java
> public FSDataOutputStream createNonRecursive(Path f,
> return this.createNonRecursive(f, FsPermission.getDefault(),
> public FSDataOutputStream createNonRecursive(Path f, FsPermission
> permission,
> throw new IOException("createNonRecursive unsupported for this
> filesystem "
>
> [root@jvyas hadoop-common]# git checkout branch-2.1.0-alpha
> Switched to branch 'branch-2.1.0-alpha'
>
> [root@jvyas hadoop-common]# cat
>
> ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
> | grep NonRec
> | grep NonRec
>
> public FSDataOutputStream createNonRecursive(Path f,
> return this.createNonRecursive(f, FsPermission.getDefault(),
> public FSDataOutputStream createNonRecursive(Path f, FsPermission
> permission,
> return createNonRecursive(f, permission,
> public FSDataOutputStream createNonRecursive(Path f, FsPermission
> permission,
> throw new IOException("createNonRecursive unsupported for this
> filesystem
>
>
> So, this (im assuming) is to retain backwards compatibility. But it when
> new API calls are made older FileSystem implementations dont work anymore.
>
> For context, here is a closely related JIRA
>
>
> https://issues.apache.org/jira/browse/HADOOP-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>
>
> On Fri, May 31, 2013 at 6:09 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > Jay, this is is more a common-dev question, as it goes beyond hdfs
> >
> > On 31 May 2013 02:07, Jay Vyas <ja...@gmail.com> wrote:
> >
> > > Hi :
> > >
> > > Are FileSystem interfaces gauranteed to retain semantics of the old
> > > FileSystem contract?
> > >
> >
> > retain the explicit semantics, yes. Implementations may change some
> > implementation details which may becomes visible
> >
> >
> > >
> > > Im noticing that the new FileSystem uses the "CreateFlag" class, which
> > > older ones did not, and I'm wondering wether the deprecation of some
> > > FileSystem methods (i.e. createNonRecursive) will eventually be
> entirely
> > > phased out.
> > >
> > > This is in reference to some issues with HBASE and the
> createNonRecursive
> > > implementation which varies across different hadoop versions.
> > >
> > >
> > can you point to the JIRAs? I presume they are append related?
> >
> >
> > > --
> > > Jay Vyas
> > > http://jayunit100.blogspot.com
> > >
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Backwards compatibility of FileSystem interface.
Posted by Jay Vyas <ja...@gmail.com>.
For context - It appears that over time the FileSystem has changed the
signature to the "createNonRecursive" method, (in 2.x there are 3
signatures, in 1.x, there are 2 signatures).
[root@jvyas hadoop-common]# git checkout branch-1.1
Switched to branch 'branch-1.1'
[root@jvyas hadoop-common]# grep NonRec
./src/core/org/apache/hadoop/fs/FileSystem.java
public FSDataOutputStream createNonRecursive(Path f,
return this.createNonRecursive(f, FsPermission.getDefault(),
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
throw new IOException("createNonRecursive unsupported for this filesystem "
[root@jvyas hadoop-common]# git checkout branch-2.1.0-alpha
Switched to branch 'branch-2.1.0-alpha'
[root@jvyas hadoop-common]# cat
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
| grep NonRec
| grep NonRec
public FSDataOutputStream createNonRecursive(Path f,
return this.createNonRecursive(f, FsPermission.getDefault(),
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
return createNonRecursive(f, permission,
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
throw new IOException("createNonRecursive unsupported for this filesystem
So, this (im assuming) is to retain backwards compatibility. But it when
new API calls are made older FileSystem implementations dont work anymore.
For context, here is a closely related JIRA
https://issues.apache.org/jira/browse/HADOOP-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
On Fri, May 31, 2013 at 6:09 AM, Steve Loughran <st...@hortonworks.com>wrote:
> Jay, this is is more a common-dev question, as it goes beyond hdfs
>
> On 31 May 2013 02:07, Jay Vyas <ja...@gmail.com> wrote:
>
> > Hi :
> >
> > Are FileSystem interfaces gauranteed to retain semantics of the old
> > FileSystem contract?
> >
>
> retain the explicit semantics, yes. Implementations may change some
> implementation details which may becomes visible
>
>
> >
> > Im noticing that the new FileSystem uses the "CreateFlag" class, which
> > older ones did not, and I'm wondering wether the deprecation of some
> > FileSystem methods (i.e. createNonRecursive) will eventually be entirely
> > phased out.
> >
> > This is in reference to some issues with HBASE and the createNonRecursive
> > implementation which varies across different hadoop versions.
> >
> >
> can you point to the JIRAs? I presume they are append related?
>
>
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
> >
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Backwards compatibility of FileSystem interface.
Posted by Steve Loughran <st...@hortonworks.com>.
Jay, this is is more a common-dev question, as it goes beyond hdfs
On 31 May 2013 02:07, Jay Vyas <ja...@gmail.com> wrote:
> Hi :
>
> Are FileSystem interfaces gauranteed to retain semantics of the old
> FileSystem contract?
>
retain the explicit semantics, yes. Implementations may change some
implementation details which may becomes visible
>
> Im noticing that the new FileSystem uses the "CreateFlag" class, which
> older ones did not, and I'm wondering wether the deprecation of some
> FileSystem methods (i.e. createNonRecursive) will eventually be entirely
> phased out.
>
> This is in reference to some issues with HBASE and the createNonRecursive
> implementation which varies across different hadoop versions.
>
>
can you point to the JIRAs? I presume they are append related?
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>