You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Aleksandr Shulman <al...@cloudera.com> on 2013/04/08 20:07:24 UTC
Declaring HBase Public API in 0.94
Hi everyone,
In light of all the conversation on compatibility, I wanted to float the
idea of documenting which Java packages, classes, and methods we want to
declare as being API compatible in 0.94.x. I'd like your input on:
1. JavaDoc vs. using AudienceInterface
2. What the javadoc notation should look like
3. Which pieces of code should be tagged
What do I mean by documenting API compatibility? That means that we suggest
the anyone building applications use specific methods because they would
continue to be both binary and RPC-compatible going forward. Any
application written, either running on a node of a cluster or on a remote
machine, would continue to work properly without recompile for all versions
of 0.94.x running on the cluster.
*Benefits:*
It would prevent developers from using calls that are subject to change.
This would give developers more confidence in using the platform, which
will encourage more development on our platform.
0.94 will still be with us for some time and I think the
better-late-than-never approach will save us pain down the road. Finally,
it would allow us to more easily verify that we are in fact API compatible.
*Can we use AudienceInterface?*
HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In the
case of 0.2x, the AudienceInterface classes were not bundled. Therefore, we
cannot expect HBase 0.94 to support it. For that reason, I think JavaDoc
might be better.
On the other hand, perhaps we might just want to bundle AudienceInterface
with 0.94 going forward? Then we can have consistent annotations in 0.94,
0.95, and 0.96 without worrying about the hadoop version.
Please correct me if I'm wrong about any of the above.
*Clarification of RPC compatibility:*
We care about RPC compatibility when we create clients that bundle their
dependency jars with them. These jars are used to form a request that is
executed on a remote machine (i.e. the cluster). If the cluster is upgraded
and no longer recognizes the command, then this will break RPC
compatibility.
*Clarification of Binary compatibility:*
We care about binary compatibility when a client is created and compiled,
and the jars on which is depends change. It should still be able to form
requests using those jars. If the cluster is upgraded and the compiled
client code cannot find a method it was depending on to be there, we break
binary compatibility. A recent example is in 0.94.2, where the return type
of HColumnDescriptor.setMaximumVersions was changed and those who upgraded
received this error:
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
HColumnDescriptor.**setMaxVersions(I)V
*What we currently have:*
We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
either adding JavaDoc or pulling in the AudienceInterface annotation.
*Suggested Javadoc language:*
@custom.94_api
*Granularity:*
Just to the class level. The native java access level (e.g. public,
protected, etc.) should indicate what should be kept compatible.
*Suggested classes:*
Here is a first-cut of things that should be declared and documented as
public APIs. This list was obtained from looking at some MapReduce over
HBase example code.
*JAVA API:*
*org.apache.hadoop.hbase (some selected classes, see below)
org.apache.hadoop.hbase.client.*
org.apache.hadoop.hbase.filter.*
org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
org.apache.hadoop.hbase.util.*
org.apache.hadoop.hbase.mapreduce.**
*REST API:
org.apache.hadoop.hbase.rest.client.**
*Thrift API:
All methods defined in:
/hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
*Selected classes in org.apache.hadoop.hbase:*
*import org.apache.hadoop.hbase.ClusterStatus;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.HRegionLocation;
import org.apache.hadoop.hbase.HServerAddress;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;*
--
Best Regards,
Aleks Shulman
847.814.5804
Cloudera
Re: Declaring HBase Public API in 0.94
Posted by Aleksandr Shulman <al...@cloudera.com>.
Hi Nick,
You're correct. This tool was designed for 0.92 and 0.94 and the most
logical place for it is in 0.94.
I plan on modifying it in trunk to be able to work with trunk code.
-Aleks S.
On Tue, Apr 16, 2013 at 2:14 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> Aleks,
>
> With the exception of the source paths [0], [1], I don't see anything that
> needs changed for 0.94. In fact, those paths look correct for 0.94 but
> wrong for trunk. I would have expected to see all of the modules broken
> out. Are you sure this generates the reports you expect when run on trunk?
>
> Thanks,
> Nick
>
> [0]:
>
> https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28#L0R38
> [1]:
>
> https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28#L0R42
>
> On Tue, Apr 16, 2013 at 1:56 PM, Aleksandr Shulman <aleks@cloudera.com
> >wrote:
>
> > Hello,
> >
> > I put together a JDiff compatibility tool that was committed to trunk.
> > Would someone be kind enough to backport it into 0.94.8?
> >
> > Here is the commit for the tool:
> >
> >
> https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28
> >
> > Here is the review:
> > https://reviews.apache.org/r/10361/
> >
> > Thanks in advance,
> >
> > Aleks S.
> >
> > On Tue, Apr 9, 2013 at 10:47 AM, Aleksandr Shulman <aleks@cloudera.com
> > >wrote:
> >
> > > I put together a tool that leverages Tom White's work.
> > > Here is a review for it: https://reviews.apache.org/r/10361/
> > >
> > > It will diff the public java api definitions of hbase in two git repos
> > and
> > > generate an HTML report. The tool will reside in the /dev-support
> folder.
> > > The documentation is inline in the file.
> > >
> > > I'd appreciate your input in how I can make it more useful and usable
> for
> > > us.
> > >
> > > Once we agree on the definitions of what classes are indeed public, we
> > can
> > > fine-tune this tool to ignore everything else.
> > >
> > > -Aleks S.
> > >
> > >
> > > On Mon, Apr 8, 2013 at 3:55 PM, Todd Lipcon <to...@cloudera.com> wrote:
> > >
> > >> The advantage of the annotations is that Tom White already did the
> work
> > >> for
> > >> jdiff to ignore non-public classes over in Hadoop land. We could
> > leverage
> > >> that work, whether we re-use the o.a.h.classification annotations or
> add
> > >> our own copies in org.apache.hbase.*.
> > >>
> > >> -Todd
> > >>
> > >> On Mon, Apr 8, 2013 at 3:08 PM, lars hofhansl <la...@apache.org>
> wrote:
> > >>
> > >> > It seems we could just generally document that:
> > >> > - no RPC incompatibilities
> > >> > - no API breaking changes to any user facing classes (now we'll pay
> > >> better
> > >> > attention to this)
> > >> > - best effort to keep coprocessor API changes backward compatible
> > >> >
> > >> > If - on the other hand - we wanted to automate API checks then we'd
> > need
> > >> > tagging (either in form of an annotation or Javadoc)
> > >> >
> > >> > +1 on the javadoc tagging if you're willing to take than on. As
> other
> > >> have
> > >> > said -1 on pulling Interface Audience in.
> > >> > Your set of classes looks good.
> > >> >
> > >> > -- Lars
> > >> >
> > >> >
> > >> >
> > >> > ________________________________
> > >> > From: Elliott Clark <ec...@apache.org>
> > >> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> > >> > Sent: Monday, April 8, 2013 1:49 PM
> > >> > Subject: Re: Declaring HBase Public API in 0.94
> > >> >
> > >> > Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility
> > was
> > >> > something that was hard won in 0.94, it would be a real shame to
> loose
> > >> that
> > >> > now.
> > >> >
> > >> >
> > >> > On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <
> > aleks@cloudera.com
> > >> > >wrote:
> > >> >
> > >> > > Hi everyone,
> > >> > >
> > >> > > In light of all the conversation on compatibility, I wanted to
> float
> > >> the
> > >> > > idea of documenting which Java packages, classes, and methods we
> > want
> > >> to
> > >> > > declare as being API compatible in 0.94.x. I'd like your input on:
> > >> > > 1. JavaDoc vs. using AudienceInterface
> > >> > > 2. What the javadoc notation should look like
> > >> > > 3. Which pieces of code should be tagged
> > >> > >
> > >> > > What do I mean by documenting API compatibility? That means that
> we
> > >> > suggest
> > >> > > the anyone building applications use specific methods because they
> > >> would
> > >> > > continue to be both binary and RPC-compatible going forward. Any
> > >> > > application written, either running on a node of a cluster or on a
> > >> remote
> > >> > > machine, would continue to work properly without recompile for all
> > >> > versions
> > >> > > of 0.94.x running on the cluster.
> > >> > >
> > >> > > *Benefits:*
> > >> > > It would prevent developers from using calls that are subject to
> > >> change.
> > >> > > This would give developers more confidence in using the platform,
> > >> which
> > >> > > will encourage more development on our platform.
> > >> > > 0.94 will still be with us for some time and I think the
> > >> > > better-late-than-never approach will save us pain down the road.
> > >> Finally,
> > >> > > it would allow us to more easily verify that we are in fact API
> > >> > compatible.
> > >> > >
> > >> > > *Can we use AudienceInterface?*
> > >> > > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and
> 2.0.x.
> > >> In
> > >> > the
> > >> > > case of 0.2x, the AudienceInterface classes were not bundled.
> > >> Therefore,
> > >> > we
> > >> > > cannot expect HBase 0.94 to support it. For that reason, I think
> > >> JavaDoc
> > >> > > might be better.
> > >> > > On the other hand, perhaps we might just want to bundle
> > >> AudienceInterface
> > >> > > with 0.94 going forward? Then we can have consistent annotations
> in
> > >> 0.94,
> > >> > > 0.95, and 0.96 without worrying about the hadoop version.
> > >> > >
> > >> > > Please correct me if I'm wrong about any of the above.
> > >> > >
> > >> > > *Clarification of RPC compatibility:*
> > >> > > We care about RPC compatibility when we create clients that bundle
> > >> their
> > >> > > dependency jars with them. These jars are used to form a request
> > that
> > >> is
> > >> > > executed on a remote machine (i.e. the cluster). If the cluster is
> > >> > upgraded
> > >> > > and no longer recognizes the command, then this will break RPC
> > >> > > compatibility.
> > >> > >
> > >> > > *Clarification of Binary compatibility:*
> > >> > > We care about binary compatibility when a client is created and
> > >> compiled,
> > >> > > and the jars on which is depends change. It should still be able
> to
> > >> form
> > >> > > requests using those jars. If the cluster is upgraded and the
> > compiled
> > >> > > client code cannot find a method it was depending on to be there,
> we
> > >> > break
> > >> > > binary compatibility. A recent example is in 0.94.2, where the
> > return
> > >> > type
> > >> > > of HColumnDescriptor.setMaximumVersions was changed and those who
> > >> > upgraded
> > >> > > received this error:
> > >> > >
> > >> > > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> > >> > > HColumnDescriptor.**setMaxVersions(I)V
> > >> > >
> > >> > > *What we currently have:*
> > >> > > We have an @audience annotation set up in 0.95/0.96. In 0.94, I
> > >> suggest
> > >> > > either adding JavaDoc or pulling in the AudienceInterface
> > annotation.
> > >> > >
> > >> > > *Suggested Javadoc language:*
> > >> > > @custom.94_api
> > >> > >
> > >> > > *Granularity:*
> > >> > > Just to the class level. The native java access level (e.g.
> public,
> > >> > > protected, etc.) should indicate what should be kept compatible.
> > >> > >
> > >> > > *Suggested classes:*
> > >> > > Here is a first-cut of things that should be declared and
> documented
> > >> as
> > >> > > public APIs. This list was obtained from looking at some MapReduce
> > >> over
> > >> > > HBase example code.
> > >> > >
> > >> > > *JAVA API:*
> > >> > >
> > >> > > *org.apache.hadoop.hbase (some selected classes, see below)
> > >> > > org.apache.hadoop.hbase.client.*
> > >> > > org.apache.hadoop.hbase.filter.*
> > >> > > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> > >> > > org.apache.hadoop.hbase.util.*
> > >> > > org.apache.hadoop.hbase.mapreduce.**
> > >> > >
> > >> > > *REST API:
> > >> > > org.apache.hadoop.hbase.rest.client.**
> > >> > >
> > >> > > *Thrift API:
> > >> > > All methods defined in:
> > >> > >
> > /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> > >> > >
> > >> > > *Selected classes in org.apache.hadoop.hbase:*
> > >> > >
> > >> > > *import org.apache.hadoop.hbase.ClusterStatus;
> > >> > > import org.apache.hadoop.hbase.HBaseConfiguration;
> > >> > > import org.apache.hadoop.hbase.HColumnDescriptor;
> > >> > > import org.apache.hadoop.hbase.HRegionInfo;
> > >> > > import org.apache.hadoop.hbase.HRegionLocation;
> > >> > > import org.apache.hadoop.hbase.HServerAddress;
> > >> > > import org.apache.hadoop.hbase.HTableDescriptor;
> > >> > > import org.apache.hadoop.hbase.KeyValue;*
> > >> > >
> > >> > > --
> > >> > > Best Regards,
> > >> > >
> > >> > > Aleks Shulman
> > >> > > 847.814.5804
> > >> > > Cloudera
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Todd Lipcon
> > >> Software Engineer, Cloudera
> > >>
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > >
> > > Aleks Shulman
> > > 847.814.5804
> > > Cloudera
> > >
> >
> >
> >
> > --
> > Best Regards,
> >
> > Aleks Shulman
> > 847.814.5804
> > Cloudera
> >
>
--
Best Regards,
Aleks Shulman
847.814.5804
Cloudera
Re: Declaring HBase Public API in 0.94
Posted by Nick Dimiduk <nd...@gmail.com>.
Aleks,
With the exception of the source paths [0], [1], I don't see anything that
needs changed for 0.94. In fact, those paths look correct for 0.94 but
wrong for trunk. I would have expected to see all of the modules broken
out. Are you sure this generates the reports you expect when run on trunk?
Thanks,
Nick
[0]:
https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28#L0R38
[1]:
https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28#L0R42
On Tue, Apr 16, 2013 at 1:56 PM, Aleksandr Shulman <al...@cloudera.com>wrote:
> Hello,
>
> I put together a JDiff compatibility tool that was committed to trunk.
> Would someone be kind enough to backport it into 0.94.8?
>
> Here is the commit for the tool:
>
> https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28
>
> Here is the review:
> https://reviews.apache.org/r/10361/
>
> Thanks in advance,
>
> Aleks S.
>
> On Tue, Apr 9, 2013 at 10:47 AM, Aleksandr Shulman <aleks@cloudera.com
> >wrote:
>
> > I put together a tool that leverages Tom White's work.
> > Here is a review for it: https://reviews.apache.org/r/10361/
> >
> > It will diff the public java api definitions of hbase in two git repos
> and
> > generate an HTML report. The tool will reside in the /dev-support folder.
> > The documentation is inline in the file.
> >
> > I'd appreciate your input in how I can make it more useful and usable for
> > us.
> >
> > Once we agree on the definitions of what classes are indeed public, we
> can
> > fine-tune this tool to ignore everything else.
> >
> > -Aleks S.
> >
> >
> > On Mon, Apr 8, 2013 at 3:55 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >> The advantage of the annotations is that Tom White already did the work
> >> for
> >> jdiff to ignore non-public classes over in Hadoop land. We could
> leverage
> >> that work, whether we re-use the o.a.h.classification annotations or add
> >> our own copies in org.apache.hbase.*.
> >>
> >> -Todd
> >>
> >> On Mon, Apr 8, 2013 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
> >>
> >> > It seems we could just generally document that:
> >> > - no RPC incompatibilities
> >> > - no API breaking changes to any user facing classes (now we'll pay
> >> better
> >> > attention to this)
> >> > - best effort to keep coprocessor API changes backward compatible
> >> >
> >> > If - on the other hand - we wanted to automate API checks then we'd
> need
> >> > tagging (either in form of an annotation or Javadoc)
> >> >
> >> > +1 on the javadoc tagging if you're willing to take than on. As other
> >> have
> >> > said -1 on pulling Interface Audience in.
> >> > Your set of classes looks good.
> >> >
> >> > -- Lars
> >> >
> >> >
> >> >
> >> > ________________________________
> >> > From: Elliott Clark <ec...@apache.org>
> >> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >> > Sent: Monday, April 8, 2013 1:49 PM
> >> > Subject: Re: Declaring HBase Public API in 0.94
> >> >
> >> > Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility
> was
> >> > something that was hard won in 0.94, it would be a real shame to loose
> >> that
> >> > now.
> >> >
> >> >
> >> > On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <
> aleks@cloudera.com
> >> > >wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > In light of all the conversation on compatibility, I wanted to float
> >> the
> >> > > idea of documenting which Java packages, classes, and methods we
> want
> >> to
> >> > > declare as being API compatible in 0.94.x. I'd like your input on:
> >> > > 1. JavaDoc vs. using AudienceInterface
> >> > > 2. What the javadoc notation should look like
> >> > > 3. Which pieces of code should be tagged
> >> > >
> >> > > What do I mean by documenting API compatibility? That means that we
> >> > suggest
> >> > > the anyone building applications use specific methods because they
> >> would
> >> > > continue to be both binary and RPC-compatible going forward. Any
> >> > > application written, either running on a node of a cluster or on a
> >> remote
> >> > > machine, would continue to work properly without recompile for all
> >> > versions
> >> > > of 0.94.x running on the cluster.
> >> > >
> >> > > *Benefits:*
> >> > > It would prevent developers from using calls that are subject to
> >> change.
> >> > > This would give developers more confidence in using the platform,
> >> which
> >> > > will encourage more development on our platform.
> >> > > 0.94 will still be with us for some time and I think the
> >> > > better-late-than-never approach will save us pain down the road.
> >> Finally,
> >> > > it would allow us to more easily verify that we are in fact API
> >> > compatible.
> >> > >
> >> > > *Can we use AudienceInterface?*
> >> > > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x.
> >> In
> >> > the
> >> > > case of 0.2x, the AudienceInterface classes were not bundled.
> >> Therefore,
> >> > we
> >> > > cannot expect HBase 0.94 to support it. For that reason, I think
> >> JavaDoc
> >> > > might be better.
> >> > > On the other hand, perhaps we might just want to bundle
> >> AudienceInterface
> >> > > with 0.94 going forward? Then we can have consistent annotations in
> >> 0.94,
> >> > > 0.95, and 0.96 without worrying about the hadoop version.
> >> > >
> >> > > Please correct me if I'm wrong about any of the above.
> >> > >
> >> > > *Clarification of RPC compatibility:*
> >> > > We care about RPC compatibility when we create clients that bundle
> >> their
> >> > > dependency jars with them. These jars are used to form a request
> that
> >> is
> >> > > executed on a remote machine (i.e. the cluster). If the cluster is
> >> > upgraded
> >> > > and no longer recognizes the command, then this will break RPC
> >> > > compatibility.
> >> > >
> >> > > *Clarification of Binary compatibility:*
> >> > > We care about binary compatibility when a client is created and
> >> compiled,
> >> > > and the jars on which is depends change. It should still be able to
> >> form
> >> > > requests using those jars. If the cluster is upgraded and the
> compiled
> >> > > client code cannot find a method it was depending on to be there, we
> >> > break
> >> > > binary compatibility. A recent example is in 0.94.2, where the
> return
> >> > type
> >> > > of HColumnDescriptor.setMaximumVersions was changed and those who
> >> > upgraded
> >> > > received this error:
> >> > >
> >> > > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> >> > > HColumnDescriptor.**setMaxVersions(I)V
> >> > >
> >> > > *What we currently have:*
> >> > > We have an @audience annotation set up in 0.95/0.96. In 0.94, I
> >> suggest
> >> > > either adding JavaDoc or pulling in the AudienceInterface
> annotation.
> >> > >
> >> > > *Suggested Javadoc language:*
> >> > > @custom.94_api
> >> > >
> >> > > *Granularity:*
> >> > > Just to the class level. The native java access level (e.g. public,
> >> > > protected, etc.) should indicate what should be kept compatible.
> >> > >
> >> > > *Suggested classes:*
> >> > > Here is a first-cut of things that should be declared and documented
> >> as
> >> > > public APIs. This list was obtained from looking at some MapReduce
> >> over
> >> > > HBase example code.
> >> > >
> >> > > *JAVA API:*
> >> > >
> >> > > *org.apache.hadoop.hbase (some selected classes, see below)
> >> > > org.apache.hadoop.hbase.client.*
> >> > > org.apache.hadoop.hbase.filter.*
> >> > > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> >> > > org.apache.hadoop.hbase.util.*
> >> > > org.apache.hadoop.hbase.mapreduce.**
> >> > >
> >> > > *REST API:
> >> > > org.apache.hadoop.hbase.rest.client.**
> >> > >
> >> > > *Thrift API:
> >> > > All methods defined in:
> >> > >
> /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> >> > >
> >> > > *Selected classes in org.apache.hadoop.hbase:*
> >> > >
> >> > > *import org.apache.hadoop.hbase.ClusterStatus;
> >> > > import org.apache.hadoop.hbase.HBaseConfiguration;
> >> > > import org.apache.hadoop.hbase.HColumnDescriptor;
> >> > > import org.apache.hadoop.hbase.HRegionInfo;
> >> > > import org.apache.hadoop.hbase.HRegionLocation;
> >> > > import org.apache.hadoop.hbase.HServerAddress;
> >> > > import org.apache.hadoop.hbase.HTableDescriptor;
> >> > > import org.apache.hadoop.hbase.KeyValue;*
> >> > >
> >> > > --
> >> > > Best Regards,
> >> > >
> >> > > Aleks Shulman
> >> > > 847.814.5804
> >> > > Cloudera
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
> >
> >
> > --
> > Best Regards,
> >
> > Aleks Shulman
> > 847.814.5804
> > Cloudera
> >
>
>
>
> --
> Best Regards,
>
> Aleks Shulman
> 847.814.5804
> Cloudera
>
Re: Declaring HBase Public API in 0.94
Posted by Aleksandr Shulman <al...@cloudera.com>.
Hello,
I put together a JDiff compatibility tool that was committed to trunk.
Would someone be kind enough to backport it into 0.94.8?
Here is the commit for the tool:
https://github.com/apache/hbase/commit/379f88f0b55b817cfa395326d5da0e3d6552ef28
Here is the review:
https://reviews.apache.org/r/10361/
Thanks in advance,
Aleks S.
On Tue, Apr 9, 2013 at 10:47 AM, Aleksandr Shulman <al...@cloudera.com>wrote:
> I put together a tool that leverages Tom White's work.
> Here is a review for it: https://reviews.apache.org/r/10361/
>
> It will diff the public java api definitions of hbase in two git repos and
> generate an HTML report. The tool will reside in the /dev-support folder.
> The documentation is inline in the file.
>
> I'd appreciate your input in how I can make it more useful and usable for
> us.
>
> Once we agree on the definitions of what classes are indeed public, we can
> fine-tune this tool to ignore everything else.
>
> -Aleks S.
>
>
> On Mon, Apr 8, 2013 at 3:55 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> The advantage of the annotations is that Tom White already did the work
>> for
>> jdiff to ignore non-public classes over in Hadoop land. We could leverage
>> that work, whether we re-use the o.a.h.classification annotations or add
>> our own copies in org.apache.hbase.*.
>>
>> -Todd
>>
>> On Mon, Apr 8, 2013 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
>>
>> > It seems we could just generally document that:
>> > - no RPC incompatibilities
>> > - no API breaking changes to any user facing classes (now we'll pay
>> better
>> > attention to this)
>> > - best effort to keep coprocessor API changes backward compatible
>> >
>> > If - on the other hand - we wanted to automate API checks then we'd need
>> > tagging (either in form of an annotation or Javadoc)
>> >
>> > +1 on the javadoc tagging if you're willing to take than on. As other
>> have
>> > said -1 on pulling Interface Audience in.
>> > Your set of classes looks good.
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ________________________________
>> > From: Elliott Clark <ec...@apache.org>
>> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> > Sent: Monday, April 8, 2013 1:49 PM
>> > Subject: Re: Declaring HBase Public API in 0.94
>> >
>> > Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
>> > something that was hard won in 0.94, it would be a real shame to loose
>> that
>> > now.
>> >
>> >
>> > On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <aleks@cloudera.com
>> > >wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > In light of all the conversation on compatibility, I wanted to float
>> the
>> > > idea of documenting which Java packages, classes, and methods we want
>> to
>> > > declare as being API compatible in 0.94.x. I'd like your input on:
>> > > 1. JavaDoc vs. using AudienceInterface
>> > > 2. What the javadoc notation should look like
>> > > 3. Which pieces of code should be tagged
>> > >
>> > > What do I mean by documenting API compatibility? That means that we
>> > suggest
>> > > the anyone building applications use specific methods because they
>> would
>> > > continue to be both binary and RPC-compatible going forward. Any
>> > > application written, either running on a node of a cluster or on a
>> remote
>> > > machine, would continue to work properly without recompile for all
>> > versions
>> > > of 0.94.x running on the cluster.
>> > >
>> > > *Benefits:*
>> > > It would prevent developers from using calls that are subject to
>> change.
>> > > This would give developers more confidence in using the platform,
>> which
>> > > will encourage more development on our platform.
>> > > 0.94 will still be with us for some time and I think the
>> > > better-late-than-never approach will save us pain down the road.
>> Finally,
>> > > it would allow us to more easily verify that we are in fact API
>> > compatible.
>> > >
>> > > *Can we use AudienceInterface?*
>> > > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x.
>> In
>> > the
>> > > case of 0.2x, the AudienceInterface classes were not bundled.
>> Therefore,
>> > we
>> > > cannot expect HBase 0.94 to support it. For that reason, I think
>> JavaDoc
>> > > might be better.
>> > > On the other hand, perhaps we might just want to bundle
>> AudienceInterface
>> > > with 0.94 going forward? Then we can have consistent annotations in
>> 0.94,
>> > > 0.95, and 0.96 without worrying about the hadoop version.
>> > >
>> > > Please correct me if I'm wrong about any of the above.
>> > >
>> > > *Clarification of RPC compatibility:*
>> > > We care about RPC compatibility when we create clients that bundle
>> their
>> > > dependency jars with them. These jars are used to form a request that
>> is
>> > > executed on a remote machine (i.e. the cluster). If the cluster is
>> > upgraded
>> > > and no longer recognizes the command, then this will break RPC
>> > > compatibility.
>> > >
>> > > *Clarification of Binary compatibility:*
>> > > We care about binary compatibility when a client is created and
>> compiled,
>> > > and the jars on which is depends change. It should still be able to
>> form
>> > > requests using those jars. If the cluster is upgraded and the compiled
>> > > client code cannot find a method it was depending on to be there, we
>> > break
>> > > binary compatibility. A recent example is in 0.94.2, where the return
>> > type
>> > > of HColumnDescriptor.setMaximumVersions was changed and those who
>> > upgraded
>> > > received this error:
>> > >
>> > > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
>> > > HColumnDescriptor.**setMaxVersions(I)V
>> > >
>> > > *What we currently have:*
>> > > We have an @audience annotation set up in 0.95/0.96. In 0.94, I
>> suggest
>> > > either adding JavaDoc or pulling in the AudienceInterface annotation.
>> > >
>> > > *Suggested Javadoc language:*
>> > > @custom.94_api
>> > >
>> > > *Granularity:*
>> > > Just to the class level. The native java access level (e.g. public,
>> > > protected, etc.) should indicate what should be kept compatible.
>> > >
>> > > *Suggested classes:*
>> > > Here is a first-cut of things that should be declared and documented
>> as
>> > > public APIs. This list was obtained from looking at some MapReduce
>> over
>> > > HBase example code.
>> > >
>> > > *JAVA API:*
>> > >
>> > > *org.apache.hadoop.hbase (some selected classes, see below)
>> > > org.apache.hadoop.hbase.client.*
>> > > org.apache.hadoop.hbase.filter.*
>> > > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
>> > > org.apache.hadoop.hbase.util.*
>> > > org.apache.hadoop.hbase.mapreduce.**
>> > >
>> > > *REST API:
>> > > org.apache.hadoop.hbase.rest.client.**
>> > >
>> > > *Thrift API:
>> > > All methods defined in:
>> > > /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
>> > >
>> > > *Selected classes in org.apache.hadoop.hbase:*
>> > >
>> > > *import org.apache.hadoop.hbase.ClusterStatus;
>> > > import org.apache.hadoop.hbase.HBaseConfiguration;
>> > > import org.apache.hadoop.hbase.HColumnDescriptor;
>> > > import org.apache.hadoop.hbase.HRegionInfo;
>> > > import org.apache.hadoop.hbase.HRegionLocation;
>> > > import org.apache.hadoop.hbase.HServerAddress;
>> > > import org.apache.hadoop.hbase.HTableDescriptor;
>> > > import org.apache.hadoop.hbase.KeyValue;*
>> > >
>> > > --
>> > > Best Regards,
>> > >
>> > > Aleks Shulman
>> > > 847.814.5804
>> > > Cloudera
>> > >
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Best Regards,
>
> Aleks Shulman
> 847.814.5804
> Cloudera
>
--
Best Regards,
Aleks Shulman
847.814.5804
Cloudera
Re: Declaring HBase Public API in 0.94
Posted by Aleksandr Shulman <al...@cloudera.com>.
I put together a tool that leverages Tom White's work.
Here is a review for it: https://reviews.apache.org/r/10361/
It will diff the public java api definitions of hbase in two git repos and
generate an HTML report. The tool will reside in the /dev-support folder.
The documentation is inline in the file.
I'd appreciate your input in how I can make it more useful and usable for
us.
Once we agree on the definitions of what classes are indeed public, we can
fine-tune this tool to ignore everything else.
-Aleks S.
On Mon, Apr 8, 2013 at 3:55 PM, Todd Lipcon <to...@cloudera.com> wrote:
> The advantage of the annotations is that Tom White already did the work for
> jdiff to ignore non-public classes over in Hadoop land. We could leverage
> that work, whether we re-use the o.a.h.classification annotations or add
> our own copies in org.apache.hbase.*.
>
> -Todd
>
> On Mon, Apr 8, 2013 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
>
> > It seems we could just generally document that:
> > - no RPC incompatibilities
> > - no API breaking changes to any user facing classes (now we'll pay
> better
> > attention to this)
> > - best effort to keep coprocessor API changes backward compatible
> >
> > If - on the other hand - we wanted to automate API checks then we'd need
> > tagging (either in form of an annotation or Javadoc)
> >
> > +1 on the javadoc tagging if you're willing to take than on. As other
> have
> > said -1 on pulling Interface Audience in.
> > Your set of classes looks good.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Elliott Clark <ec...@apache.org>
> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> > Sent: Monday, April 8, 2013 1:49 PM
> > Subject: Re: Declaring HBase Public API in 0.94
> >
> > Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
> > something that was hard won in 0.94, it would be a real shame to loose
> that
> > now.
> >
> >
> > On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <aleks@cloudera.com
> > >wrote:
> >
> > > Hi everyone,
> > >
> > > In light of all the conversation on compatibility, I wanted to float
> the
> > > idea of documenting which Java packages, classes, and methods we want
> to
> > > declare as being API compatible in 0.94.x. I'd like your input on:
> > > 1. JavaDoc vs. using AudienceInterface
> > > 2. What the javadoc notation should look like
> > > 3. Which pieces of code should be tagged
> > >
> > > What do I mean by documenting API compatibility? That means that we
> > suggest
> > > the anyone building applications use specific methods because they
> would
> > > continue to be both binary and RPC-compatible going forward. Any
> > > application written, either running on a node of a cluster or on a
> remote
> > > machine, would continue to work properly without recompile for all
> > versions
> > > of 0.94.x running on the cluster.
> > >
> > > *Benefits:*
> > > It would prevent developers from using calls that are subject to
> change.
> > > This would give developers more confidence in using the platform, which
> > > will encourage more development on our platform.
> > > 0.94 will still be with us for some time and I think the
> > > better-late-than-never approach will save us pain down the road.
> Finally,
> > > it would allow us to more easily verify that we are in fact API
> > compatible.
> > >
> > > *Can we use AudienceInterface?*
> > > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In
> > the
> > > case of 0.2x, the AudienceInterface classes were not bundled.
> Therefore,
> > we
> > > cannot expect HBase 0.94 to support it. For that reason, I think
> JavaDoc
> > > might be better.
> > > On the other hand, perhaps we might just want to bundle
> AudienceInterface
> > > with 0.94 going forward? Then we can have consistent annotations in
> 0.94,
> > > 0.95, and 0.96 without worrying about the hadoop version.
> > >
> > > Please correct me if I'm wrong about any of the above.
> > >
> > > *Clarification of RPC compatibility:*
> > > We care about RPC compatibility when we create clients that bundle
> their
> > > dependency jars with them. These jars are used to form a request that
> is
> > > executed on a remote machine (i.e. the cluster). If the cluster is
> > upgraded
> > > and no longer recognizes the command, then this will break RPC
> > > compatibility.
> > >
> > > *Clarification of Binary compatibility:*
> > > We care about binary compatibility when a client is created and
> compiled,
> > > and the jars on which is depends change. It should still be able to
> form
> > > requests using those jars. If the cluster is upgraded and the compiled
> > > client code cannot find a method it was depending on to be there, we
> > break
> > > binary compatibility. A recent example is in 0.94.2, where the return
> > type
> > > of HColumnDescriptor.setMaximumVersions was changed and those who
> > upgraded
> > > received this error:
> > >
> > > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> > > HColumnDescriptor.**setMaxVersions(I)V
> > >
> > > *What we currently have:*
> > > We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
> > > either adding JavaDoc or pulling in the AudienceInterface annotation.
> > >
> > > *Suggested Javadoc language:*
> > > @custom.94_api
> > >
> > > *Granularity:*
> > > Just to the class level. The native java access level (e.g. public,
> > > protected, etc.) should indicate what should be kept compatible.
> > >
> > > *Suggested classes:*
> > > Here is a first-cut of things that should be declared and documented as
> > > public APIs. This list was obtained from looking at some MapReduce over
> > > HBase example code.
> > >
> > > *JAVA API:*
> > >
> > > *org.apache.hadoop.hbase (some selected classes, see below)
> > > org.apache.hadoop.hbase.client.*
> > > org.apache.hadoop.hbase.filter.*
> > > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> > > org.apache.hadoop.hbase.util.*
> > > org.apache.hadoop.hbase.mapreduce.**
> > >
> > > *REST API:
> > > org.apache.hadoop.hbase.rest.client.**
> > >
> > > *Thrift API:
> > > All methods defined in:
> > > /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> > >
> > > *Selected classes in org.apache.hadoop.hbase:*
> > >
> > > *import org.apache.hadoop.hbase.ClusterStatus;
> > > import org.apache.hadoop.hbase.HBaseConfiguration;
> > > import org.apache.hadoop.hbase.HColumnDescriptor;
> > > import org.apache.hadoop.hbase.HRegionInfo;
> > > import org.apache.hadoop.hbase.HRegionLocation;
> > > import org.apache.hadoop.hbase.HServerAddress;
> > > import org.apache.hadoop.hbase.HTableDescriptor;
> > > import org.apache.hadoop.hbase.KeyValue;*
> > >
> > > --
> > > Best Regards,
> > >
> > > Aleks Shulman
> > > 847.814.5804
> > > Cloudera
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Best Regards,
Aleks Shulman
847.814.5804
Cloudera
Re: Declaring HBase Public API in 0.94
Posted by Todd Lipcon <to...@cloudera.com>.
The advantage of the annotations is that Tom White already did the work for
jdiff to ignore non-public classes over in Hadoop land. We could leverage
that work, whether we re-use the o.a.h.classification annotations or add
our own copies in org.apache.hbase.*.
-Todd
On Mon, Apr 8, 2013 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
> It seems we could just generally document that:
> - no RPC incompatibilities
> - no API breaking changes to any user facing classes (now we'll pay better
> attention to this)
> - best effort to keep coprocessor API changes backward compatible
>
> If - on the other hand - we wanted to automate API checks then we'd need
> tagging (either in form of an annotation or Javadoc)
>
> +1 on the javadoc tagging if you're willing to take than on. As other have
> said -1 on pulling Interface Audience in.
> Your set of classes looks good.
>
> -- Lars
>
>
>
> ________________________________
> From: Elliott Clark <ec...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Monday, April 8, 2013 1:49 PM
> Subject: Re: Declaring HBase Public API in 0.94
>
> Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
> something that was hard won in 0.94, it would be a real shame to loose that
> now.
>
>
> On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <aleks@cloudera.com
> >wrote:
>
> > Hi everyone,
> >
> > In light of all the conversation on compatibility, I wanted to float the
> > idea of documenting which Java packages, classes, and methods we want to
> > declare as being API compatible in 0.94.x. I'd like your input on:
> > 1. JavaDoc vs. using AudienceInterface
> > 2. What the javadoc notation should look like
> > 3. Which pieces of code should be tagged
> >
> > What do I mean by documenting API compatibility? That means that we
> suggest
> > the anyone building applications use specific methods because they would
> > continue to be both binary and RPC-compatible going forward. Any
> > application written, either running on a node of a cluster or on a remote
> > machine, would continue to work properly without recompile for all
> versions
> > of 0.94.x running on the cluster.
> >
> > *Benefits:*
> > It would prevent developers from using calls that are subject to change.
> > This would give developers more confidence in using the platform, which
> > will encourage more development on our platform.
> > 0.94 will still be with us for some time and I think the
> > better-late-than-never approach will save us pain down the road. Finally,
> > it would allow us to more easily verify that we are in fact API
> compatible.
> >
> > *Can we use AudienceInterface?*
> > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In
> the
> > case of 0.2x, the AudienceInterface classes were not bundled. Therefore,
> we
> > cannot expect HBase 0.94 to support it. For that reason, I think JavaDoc
> > might be better.
> > On the other hand, perhaps we might just want to bundle AudienceInterface
> > with 0.94 going forward? Then we can have consistent annotations in 0.94,
> > 0.95, and 0.96 without worrying about the hadoop version.
> >
> > Please correct me if I'm wrong about any of the above.
> >
> > *Clarification of RPC compatibility:*
> > We care about RPC compatibility when we create clients that bundle their
> > dependency jars with them. These jars are used to form a request that is
> > executed on a remote machine (i.e. the cluster). If the cluster is
> upgraded
> > and no longer recognizes the command, then this will break RPC
> > compatibility.
> >
> > *Clarification of Binary compatibility:*
> > We care about binary compatibility when a client is created and compiled,
> > and the jars on which is depends change. It should still be able to form
> > requests using those jars. If the cluster is upgraded and the compiled
> > client code cannot find a method it was depending on to be there, we
> break
> > binary compatibility. A recent example is in 0.94.2, where the return
> type
> > of HColumnDescriptor.setMaximumVersions was changed and those who
> upgraded
> > received this error:
> >
> > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> > HColumnDescriptor.**setMaxVersions(I)V
> >
> > *What we currently have:*
> > We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
> > either adding JavaDoc or pulling in the AudienceInterface annotation.
> >
> > *Suggested Javadoc language:*
> > @custom.94_api
> >
> > *Granularity:*
> > Just to the class level. The native java access level (e.g. public,
> > protected, etc.) should indicate what should be kept compatible.
> >
> > *Suggested classes:*
> > Here is a first-cut of things that should be declared and documented as
> > public APIs. This list was obtained from looking at some MapReduce over
> > HBase example code.
> >
> > *JAVA API:*
> >
> > *org.apache.hadoop.hbase (some selected classes, see below)
> > org.apache.hadoop.hbase.client.*
> > org.apache.hadoop.hbase.filter.*
> > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> > org.apache.hadoop.hbase.util.*
> > org.apache.hadoop.hbase.mapreduce.**
> >
> > *REST API:
> > org.apache.hadoop.hbase.rest.client.**
> >
> > *Thrift API:
> > All methods defined in:
> > /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> >
> > *Selected classes in org.apache.hadoop.hbase:*
> >
> > *import org.apache.hadoop.hbase.ClusterStatus;
> > import org.apache.hadoop.hbase.HBaseConfiguration;
> > import org.apache.hadoop.hbase.HColumnDescriptor;
> > import org.apache.hadoop.hbase.HRegionInfo;
> > import org.apache.hadoop.hbase.HRegionLocation;
> > import org.apache.hadoop.hbase.HServerAddress;
> > import org.apache.hadoop.hbase.HTableDescriptor;
> > import org.apache.hadoop.hbase.KeyValue;*
> >
> > --
> > Best Regards,
> >
> > Aleks Shulman
> > 847.814.5804
> > Cloudera
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Declaring HBase Public API in 0.94
Posted by lars hofhansl <la...@apache.org>.
It seems we could just generally document that:
- no RPC incompatibilities
- no API breaking changes to any user facing classes (now we'll pay better attention to this)
- best effort to keep coprocessor API changes backward compatible
If - on the other hand - we wanted to automate API checks then we'd need tagging (either in form of an annotation or Javadoc)
+1 on the javadoc tagging if you're willing to take than on. As other have said -1 on pulling Interface Audience in.
Your set of classes looks good.
-- Lars
________________________________
From: Elliott Clark <ec...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Monday, April 8, 2013 1:49 PM
Subject: Re: Declaring HBase Public API in 0.94
Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
something that was hard won in 0.94, it would be a real shame to loose that
now.
On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <al...@cloudera.com>wrote:
> Hi everyone,
>
> In light of all the conversation on compatibility, I wanted to float the
> idea of documenting which Java packages, classes, and methods we want to
> declare as being API compatible in 0.94.x. I'd like your input on:
> 1. JavaDoc vs. using AudienceInterface
> 2. What the javadoc notation should look like
> 3. Which pieces of code should be tagged
>
> What do I mean by documenting API compatibility? That means that we suggest
> the anyone building applications use specific methods because they would
> continue to be both binary and RPC-compatible going forward. Any
> application written, either running on a node of a cluster or on a remote
> machine, would continue to work properly without recompile for all versions
> of 0.94.x running on the cluster.
>
> *Benefits:*
> It would prevent developers from using calls that are subject to change.
> This would give developers more confidence in using the platform, which
> will encourage more development on our platform.
> 0.94 will still be with us for some time and I think the
> better-late-than-never approach will save us pain down the road. Finally,
> it would allow us to more easily verify that we are in fact API compatible.
>
> *Can we use AudienceInterface?*
> HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In the
> case of 0.2x, the AudienceInterface classes were not bundled. Therefore, we
> cannot expect HBase 0.94 to support it. For that reason, I think JavaDoc
> might be better.
> On the other hand, perhaps we might just want to bundle AudienceInterface
> with 0.94 going forward? Then we can have consistent annotations in 0.94,
> 0.95, and 0.96 without worrying about the hadoop version.
>
> Please correct me if I'm wrong about any of the above.
>
> *Clarification of RPC compatibility:*
> We care about RPC compatibility when we create clients that bundle their
> dependency jars with them. These jars are used to form a request that is
> executed on a remote machine (i.e. the cluster). If the cluster is upgraded
> and no longer recognizes the command, then this will break RPC
> compatibility.
>
> *Clarification of Binary compatibility:*
> We care about binary compatibility when a client is created and compiled,
> and the jars on which is depends change. It should still be able to form
> requests using those jars. If the cluster is upgraded and the compiled
> client code cannot find a method it was depending on to be there, we break
> binary compatibility. A recent example is in 0.94.2, where the return type
> of HColumnDescriptor.setMaximumVersions was changed and those who upgraded
> received this error:
>
> java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> HColumnDescriptor.**setMaxVersions(I)V
>
> *What we currently have:*
> We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
> either adding JavaDoc or pulling in the AudienceInterface annotation.
>
> *Suggested Javadoc language:*
> @custom.94_api
>
> *Granularity:*
> Just to the class level. The native java access level (e.g. public,
> protected, etc.) should indicate what should be kept compatible.
>
> *Suggested classes:*
> Here is a first-cut of things that should be declared and documented as
> public APIs. This list was obtained from looking at some MapReduce over
> HBase example code.
>
> *JAVA API:*
>
> *org.apache.hadoop.hbase (some selected classes, see below)
> org.apache.hadoop.hbase.client.*
> org.apache.hadoop.hbase.filter.*
> org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> org.apache.hadoop.hbase.util.*
> org.apache.hadoop.hbase.mapreduce.**
>
> *REST API:
> org.apache.hadoop.hbase.rest.client.**
>
> *Thrift API:
> All methods defined in:
> /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
>
> *Selected classes in org.apache.hadoop.hbase:*
>
> *import org.apache.hadoop.hbase.ClusterStatus;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HColumnDescriptor;
> import org.apache.hadoop.hbase.HRegionInfo;
> import org.apache.hadoop.hbase.HRegionLocation;
> import org.apache.hadoop.hbase.HServerAddress;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.KeyValue;*
>
> --
> Best Regards,
>
> Aleks Shulman
> 847.814.5804
> Cloudera
>
Re: Declaring HBase Public API in 0.94
Posted by Enis Söztutar <en...@gmail.com>.
> Given that it's only annotations, you also may be able to pull in the
Hadoop 2 annotations package without all of common (I believe it's a
separate maven artifact)
I fear that will cause even more problems, since 0.94 will still be run
against 0.23, and 2.x. I would vote for bringing the annotations in if we
would go that route.
Enis
On Mon, Apr 8, 2013 at 2:50 PM, Todd Lipcon <to...@cloudera.com> wrote:
> There's always the option of duplicating those annotations into the HBase
> package if you want to annotate 0.94 without depending on Hadoop 2.
>
> Given that it's only annotations, you also may be able to pull in the
> Hadoop 2 annotations package without all of common (I believe it's a
> separate maven artifact)
>
> -Todd
>
> On Mon, Apr 8, 2013 at 2:24 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Keeping 0.2x compatibility was something that was hard won in 0.94
> >
> > True. We should maintain this status.
> >
> > On Mon, Apr 8, 2013 at 1:49 PM, Elliott Clark <ec...@apache.org> wrote:
> >
> > > Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility
> was
> > > something that was hard won in 0.94, it would be a real shame to loose
> > that
> > > now.
> > >
> > >
> > > On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <aleks@cloudera.com
> > > >wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > In light of all the conversation on compatibility, I wanted to float
> > the
> > > > idea of documenting which Java packages, classes, and methods we want
> > to
> > > > declare as being API compatible in 0.94.x. I'd like your input on:
> > > > 1. JavaDoc vs. using AudienceInterface
> > > > 2. What the javadoc notation should look like
> > > > 3. Which pieces of code should be tagged
> > > >
> > > > What do I mean by documenting API compatibility? That means that we
> > > suggest
> > > > the anyone building applications use specific methods because they
> > would
> > > > continue to be both binary and RPC-compatible going forward. Any
> > > > application written, either running on a node of a cluster or on a
> > remote
> > > > machine, would continue to work properly without recompile for all
> > > versions
> > > > of 0.94.x running on the cluster.
> > > >
> > > > *Benefits:*
> > > > It would prevent developers from using calls that are subject to
> > change.
> > > > This would give developers more confidence in using the platform,
> which
> > > > will encourage more development on our platform.
> > > > 0.94 will still be with us for some time and I think the
> > > > better-late-than-never approach will save us pain down the road.
> > Finally,
> > > > it would allow us to more easily verify that we are in fact API
> > > compatible.
> > > >
> > > > *Can we use AudienceInterface?*
> > > > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x.
> In
> > > the
> > > > case of 0.2x, the AudienceInterface classes were not bundled.
> > Therefore,
> > > we
> > > > cannot expect HBase 0.94 to support it. For that reason, I think
> > JavaDoc
> > > > might be better.
> > > > On the other hand, perhaps we might just want to bundle
> > AudienceInterface
> > > > with 0.94 going forward? Then we can have consistent annotations in
> > 0.94,
> > > > 0.95, and 0.96 without worrying about the hadoop version.
> > > >
> > > > Please correct me if I'm wrong about any of the above.
> > > >
> > > > *Clarification of RPC compatibility:*
> > > > We care about RPC compatibility when we create clients that bundle
> > their
> > > > dependency jars with them. These jars are used to form a request that
> > is
> > > > executed on a remote machine (i.e. the cluster). If the cluster is
> > > upgraded
> > > > and no longer recognizes the command, then this will break RPC
> > > > compatibility.
> > > >
> > > > *Clarification of Binary compatibility:*
> > > > We care about binary compatibility when a client is created and
> > compiled,
> > > > and the jars on which is depends change. It should still be able to
> > form
> > > > requests using those jars. If the cluster is upgraded and the
> compiled
> > > > client code cannot find a method it was depending on to be there, we
> > > break
> > > > binary compatibility. A recent example is in 0.94.2, where the return
> > > type
> > > > of HColumnDescriptor.setMaximumVersions was changed and those who
> > > upgraded
> > > > received this error:
> > > >
> > > > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> > > > HColumnDescriptor.**setMaxVersions(I)V
> > > >
> > > > *What we currently have:*
> > > > We have an @audience annotation set up in 0.95/0.96. In 0.94, I
> suggest
> > > > either adding JavaDoc or pulling in the AudienceInterface annotation.
> > > >
> > > > *Suggested Javadoc language:*
> > > > @custom.94_api
> > > >
> > > > *Granularity:*
> > > > Just to the class level. The native java access level (e.g. public,
> > > > protected, etc.) should indicate what should be kept compatible.
> > > >
> > > > *Suggested classes:*
> > > > Here is a first-cut of things that should be declared and documented
> as
> > > > public APIs. This list was obtained from looking at some MapReduce
> over
> > > > HBase example code.
> > > >
> > > > *JAVA API:*
> > > >
> > > > *org.apache.hadoop.hbase (some selected classes, see below)
> > > > org.apache.hadoop.hbase.client.*
> > > > org.apache.hadoop.hbase.filter.*
> > > > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> > > > org.apache.hadoop.hbase.util.*
> > > > org.apache.hadoop.hbase.mapreduce.**
> > > >
> > > > *REST API:
> > > > org.apache.hadoop.hbase.rest.client.**
> > > >
> > > > *Thrift API:
> > > > All methods defined in:
> > > >
> /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> > > >
> > > > *Selected classes in org.apache.hadoop.hbase:*
> > > >
> > > > *import org.apache.hadoop.hbase.ClusterStatus;
> > > > import org.apache.hadoop.hbase.HBaseConfiguration;
> > > > import org.apache.hadoop.hbase.HColumnDescriptor;
> > > > import org.apache.hadoop.hbase.HRegionInfo;
> > > > import org.apache.hadoop.hbase.HRegionLocation;
> > > > import org.apache.hadoop.hbase.HServerAddress;
> > > > import org.apache.hadoop.hbase.HTableDescriptor;
> > > > import org.apache.hadoop.hbase.KeyValue;*
> > > >
> > > > --
> > > > Best Regards,
> > > >
> > > > Aleks Shulman
> > > > 847.814.5804
> > > > Cloudera
> > > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Declaring HBase Public API in 0.94
Posted by Todd Lipcon <to...@cloudera.com>.
There's always the option of duplicating those annotations into the HBase
package if you want to annotate 0.94 without depending on Hadoop 2.
Given that it's only annotations, you also may be able to pull in the
Hadoop 2 annotations package without all of common (I believe it's a
separate maven artifact)
-Todd
On Mon, Apr 8, 2013 at 2:24 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Keeping 0.2x compatibility was something that was hard won in 0.94
>
> True. We should maintain this status.
>
> On Mon, Apr 8, 2013 at 1:49 PM, Elliott Clark <ec...@apache.org> wrote:
>
> > Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
> > something that was hard won in 0.94, it would be a real shame to loose
> that
> > now.
> >
> >
> > On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <aleks@cloudera.com
> > >wrote:
> >
> > > Hi everyone,
> > >
> > > In light of all the conversation on compatibility, I wanted to float
> the
> > > idea of documenting which Java packages, classes, and methods we want
> to
> > > declare as being API compatible in 0.94.x. I'd like your input on:
> > > 1. JavaDoc vs. using AudienceInterface
> > > 2. What the javadoc notation should look like
> > > 3. Which pieces of code should be tagged
> > >
> > > What do I mean by documenting API compatibility? That means that we
> > suggest
> > > the anyone building applications use specific methods because they
> would
> > > continue to be both binary and RPC-compatible going forward. Any
> > > application written, either running on a node of a cluster or on a
> remote
> > > machine, would continue to work properly without recompile for all
> > versions
> > > of 0.94.x running on the cluster.
> > >
> > > *Benefits:*
> > > It would prevent developers from using calls that are subject to
> change.
> > > This would give developers more confidence in using the platform, which
> > > will encourage more development on our platform.
> > > 0.94 will still be with us for some time and I think the
> > > better-late-than-never approach will save us pain down the road.
> Finally,
> > > it would allow us to more easily verify that we are in fact API
> > compatible.
> > >
> > > *Can we use AudienceInterface?*
> > > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In
> > the
> > > case of 0.2x, the AudienceInterface classes were not bundled.
> Therefore,
> > we
> > > cannot expect HBase 0.94 to support it. For that reason, I think
> JavaDoc
> > > might be better.
> > > On the other hand, perhaps we might just want to bundle
> AudienceInterface
> > > with 0.94 going forward? Then we can have consistent annotations in
> 0.94,
> > > 0.95, and 0.96 without worrying about the hadoop version.
> > >
> > > Please correct me if I'm wrong about any of the above.
> > >
> > > *Clarification of RPC compatibility:*
> > > We care about RPC compatibility when we create clients that bundle
> their
> > > dependency jars with them. These jars are used to form a request that
> is
> > > executed on a remote machine (i.e. the cluster). If the cluster is
> > upgraded
> > > and no longer recognizes the command, then this will break RPC
> > > compatibility.
> > >
> > > *Clarification of Binary compatibility:*
> > > We care about binary compatibility when a client is created and
> compiled,
> > > and the jars on which is depends change. It should still be able to
> form
> > > requests using those jars. If the cluster is upgraded and the compiled
> > > client code cannot find a method it was depending on to be there, we
> > break
> > > binary compatibility. A recent example is in 0.94.2, where the return
> > type
> > > of HColumnDescriptor.setMaximumVersions was changed and those who
> > upgraded
> > > received this error:
> > >
> > > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> > > HColumnDescriptor.**setMaxVersions(I)V
> > >
> > > *What we currently have:*
> > > We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
> > > either adding JavaDoc or pulling in the AudienceInterface annotation.
> > >
> > > *Suggested Javadoc language:*
> > > @custom.94_api
> > >
> > > *Granularity:*
> > > Just to the class level. The native java access level (e.g. public,
> > > protected, etc.) should indicate what should be kept compatible.
> > >
> > > *Suggested classes:*
> > > Here is a first-cut of things that should be declared and documented as
> > > public APIs. This list was obtained from looking at some MapReduce over
> > > HBase example code.
> > >
> > > *JAVA API:*
> > >
> > > *org.apache.hadoop.hbase (some selected classes, see below)
> > > org.apache.hadoop.hbase.client.*
> > > org.apache.hadoop.hbase.filter.*
> > > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> > > org.apache.hadoop.hbase.util.*
> > > org.apache.hadoop.hbase.mapreduce.**
> > >
> > > *REST API:
> > > org.apache.hadoop.hbase.rest.client.**
> > >
> > > *Thrift API:
> > > All methods defined in:
> > > /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> > >
> > > *Selected classes in org.apache.hadoop.hbase:*
> > >
> > > *import org.apache.hadoop.hbase.ClusterStatus;
> > > import org.apache.hadoop.hbase.HBaseConfiguration;
> > > import org.apache.hadoop.hbase.HColumnDescriptor;
> > > import org.apache.hadoop.hbase.HRegionInfo;
> > > import org.apache.hadoop.hbase.HRegionLocation;
> > > import org.apache.hadoop.hbase.HServerAddress;
> > > import org.apache.hadoop.hbase.HTableDescriptor;
> > > import org.apache.hadoop.hbase.KeyValue;*
> > >
> > > --
> > > Best Regards,
> > >
> > > Aleks Shulman
> > > 847.814.5804
> > > Cloudera
> > >
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Declaring HBase Public API in 0.94
Posted by Ted Yu <yu...@gmail.com>.
bq. Keeping 0.2x compatibility was something that was hard won in 0.94
True. We should maintain this status.
On Mon, Apr 8, 2013 at 1:49 PM, Elliott Clark <ec...@apache.org> wrote:
> Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
> something that was hard won in 0.94, it would be a real shame to loose that
> now.
>
>
> On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <aleks@cloudera.com
> >wrote:
>
> > Hi everyone,
> >
> > In light of all the conversation on compatibility, I wanted to float the
> > idea of documenting which Java packages, classes, and methods we want to
> > declare as being API compatible in 0.94.x. I'd like your input on:
> > 1. JavaDoc vs. using AudienceInterface
> > 2. What the javadoc notation should look like
> > 3. Which pieces of code should be tagged
> >
> > What do I mean by documenting API compatibility? That means that we
> suggest
> > the anyone building applications use specific methods because they would
> > continue to be both binary and RPC-compatible going forward. Any
> > application written, either running on a node of a cluster or on a remote
> > machine, would continue to work properly without recompile for all
> versions
> > of 0.94.x running on the cluster.
> >
> > *Benefits:*
> > It would prevent developers from using calls that are subject to change.
> > This would give developers more confidence in using the platform, which
> > will encourage more development on our platform.
> > 0.94 will still be with us for some time and I think the
> > better-late-than-never approach will save us pain down the road. Finally,
> > it would allow us to more easily verify that we are in fact API
> compatible.
> >
> > *Can we use AudienceInterface?*
> > HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In
> the
> > case of 0.2x, the AudienceInterface classes were not bundled. Therefore,
> we
> > cannot expect HBase 0.94 to support it. For that reason, I think JavaDoc
> > might be better.
> > On the other hand, perhaps we might just want to bundle AudienceInterface
> > with 0.94 going forward? Then we can have consistent annotations in 0.94,
> > 0.95, and 0.96 without worrying about the hadoop version.
> >
> > Please correct me if I'm wrong about any of the above.
> >
> > *Clarification of RPC compatibility:*
> > We care about RPC compatibility when we create clients that bundle their
> > dependency jars with them. These jars are used to form a request that is
> > executed on a remote machine (i.e. the cluster). If the cluster is
> upgraded
> > and no longer recognizes the command, then this will break RPC
> > compatibility.
> >
> > *Clarification of Binary compatibility:*
> > We care about binary compatibility when a client is created and compiled,
> > and the jars on which is depends change. It should still be able to form
> > requests using those jars. If the cluster is upgraded and the compiled
> > client code cannot find a method it was depending on to be there, we
> break
> > binary compatibility. A recent example is in 0.94.2, where the return
> type
> > of HColumnDescriptor.setMaximumVersions was changed and those who
> upgraded
> > received this error:
> >
> > java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> > HColumnDescriptor.**setMaxVersions(I)V
> >
> > *What we currently have:*
> > We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
> > either adding JavaDoc or pulling in the AudienceInterface annotation.
> >
> > *Suggested Javadoc language:*
> > @custom.94_api
> >
> > *Granularity:*
> > Just to the class level. The native java access level (e.g. public,
> > protected, etc.) should indicate what should be kept compatible.
> >
> > *Suggested classes:*
> > Here is a first-cut of things that should be declared and documented as
> > public APIs. This list was obtained from looking at some MapReduce over
> > HBase example code.
> >
> > *JAVA API:*
> >
> > *org.apache.hadoop.hbase (some selected classes, see below)
> > org.apache.hadoop.hbase.client.*
> > org.apache.hadoop.hbase.filter.*
> > org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> > org.apache.hadoop.hbase.util.*
> > org.apache.hadoop.hbase.mapreduce.**
> >
> > *REST API:
> > org.apache.hadoop.hbase.rest.client.**
> >
> > *Thrift API:
> > All methods defined in:
> > /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
> >
> > *Selected classes in org.apache.hadoop.hbase:*
> >
> > *import org.apache.hadoop.hbase.ClusterStatus;
> > import org.apache.hadoop.hbase.HBaseConfiguration;
> > import org.apache.hadoop.hbase.HColumnDescriptor;
> > import org.apache.hadoop.hbase.HRegionInfo;
> > import org.apache.hadoop.hbase.HRegionLocation;
> > import org.apache.hadoop.hbase.HServerAddress;
> > import org.apache.hadoop.hbase.HTableDescriptor;
> > import org.apache.hadoop.hbase.KeyValue;*
> >
> > --
> > Best Regards,
> >
> > Aleks Shulman
> > 847.814.5804
> > Cloudera
> >
>
Re: Declaring HBase Public API in 0.94
Posted by Elliott Clark <ec...@apache.org>.
Please don't pull in @InterfaceAudience. Keeping 0.2x compatibility was
something that was hard won in 0.94, it would be a real shame to loose that
now.
On Mon, Apr 8, 2013 at 11:07 AM, Aleksandr Shulman <al...@cloudera.com>wrote:
> Hi everyone,
>
> In light of all the conversation on compatibility, I wanted to float the
> idea of documenting which Java packages, classes, and methods we want to
> declare as being API compatible in 0.94.x. I'd like your input on:
> 1. JavaDoc vs. using AudienceInterface
> 2. What the javadoc notation should look like
> 3. Which pieces of code should be tagged
>
> What do I mean by documenting API compatibility? That means that we suggest
> the anyone building applications use specific methods because they would
> continue to be both binary and RPC-compatible going forward. Any
> application written, either running on a node of a cluster or on a remote
> machine, would continue to work properly without recompile for all versions
> of 0.94.x running on the cluster.
>
> *Benefits:*
> It would prevent developers from using calls that are subject to change.
> This would give developers more confidence in using the platform, which
> will encourage more development on our platform.
> 0.94 will still be with us for some time and I think the
> better-late-than-never approach will save us pain down the road. Finally,
> it would allow us to more easily verify that we are in fact API compatible.
>
> *Can we use AudienceInterface?*
> HBase 0.94 can be compiled against both hadoop 0.2x, 1.x, and 2.0.x. In the
> case of 0.2x, the AudienceInterface classes were not bundled. Therefore, we
> cannot expect HBase 0.94 to support it. For that reason, I think JavaDoc
> might be better.
> On the other hand, perhaps we might just want to bundle AudienceInterface
> with 0.94 going forward? Then we can have consistent annotations in 0.94,
> 0.95, and 0.96 without worrying about the hadoop version.
>
> Please correct me if I'm wrong about any of the above.
>
> *Clarification of RPC compatibility:*
> We care about RPC compatibility when we create clients that bundle their
> dependency jars with them. These jars are used to form a request that is
> executed on a remote machine (i.e. the cluster). If the cluster is upgraded
> and no longer recognizes the command, then this will break RPC
> compatibility.
>
> *Clarification of Binary compatibility:*
> We care about binary compatibility when a client is created and compiled,
> and the jars on which is depends change. It should still be able to form
> requests using those jars. If the cluster is upgraded and the compiled
> client code cannot find a method it was depending on to be there, we break
> binary compatibility. A recent example is in 0.94.2, where the return type
> of HColumnDescriptor.setMaximumVersions was changed and those who upgraded
> received this error:
>
> java.lang.NoSuchMethodError: org.apache.hadoop.hbase.**
> HColumnDescriptor.**setMaxVersions(I)V
>
> *What we currently have:*
> We have an @audience annotation set up in 0.95/0.96. In 0.94, I suggest
> either adding JavaDoc or pulling in the AudienceInterface annotation.
>
> *Suggested Javadoc language:*
> @custom.94_api
>
> *Granularity:*
> Just to the class level. The native java access level (e.g. public,
> protected, etc.) should indicate what should be kept compatible.
>
> *Suggested classes:*
> Here is a first-cut of things that should be declared and documented as
> public APIs. This list was obtained from looking at some MapReduce over
> HBase example code.
>
> *JAVA API:*
>
> *org.apache.hadoop.hbase (some selected classes, see below)
> org.apache.hadoop.hbase.client.*
> org.apache.hadoop.hbase.filter.*
> org.apache.hadoop.hbase.io.hfile.Compression.Algorithm
> org.apache.hadoop.hbase.util.*
> org.apache.hadoop.hbase.mapreduce.**
>
> *REST API:
> org.apache.hadoop.hbase.rest.client.**
>
> *Thrift API:
> All methods defined in:
> /hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift*
>
> *Selected classes in org.apache.hadoop.hbase:*
>
> *import org.apache.hadoop.hbase.ClusterStatus;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HColumnDescriptor;
> import org.apache.hadoop.hbase.HRegionInfo;
> import org.apache.hadoop.hbase.HRegionLocation;
> import org.apache.hadoop.hbase.HServerAddress;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.KeyValue;*
>
> --
> Best Regards,
>
> Aleks Shulman
> 847.814.5804
> Cloudera
>