You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sanjay Radia (JIRA)" <ji...@apache.org> on 2009/01/16 21:59:59 UTC
[jira] Created: (HADOOP-5073) Hadoop 1.0 Interface Classification -
scope (visibility - public/private) and stability
Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
---------------------------------------------------------------------------------------
Key: HADOOP-5073
URL: https://issues.apache.org/jira/browse/HADOOP-5073
Project: Hadoop Core
Issue Type: Sub-task
Reporter: Sanjay Radia
Assignee: Sanjay Radia
This jira proposes an interface classification for hadoop interfaces.
The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
h2. Interface Taxonomy - Scope & Stability Classification
The interface taxonomy classification provided here is for guidance to developers and users of interfaces.
The classification guides a developer to declare the scope (or targeted audience or users) of an interface and also its stability.
* *Benefits to the user of an interface*: Knows which interfaces to use or not use and their stability.
* *Benefits to the developer*: to prevent accidental changes of interfaces and hence accidental impact on users or other components or system. This is particularly useful in large systems with many developers who may not all have a shared state/history of the project.
This classification was derived from a taxonomy used inside Yahoo and
from the OpenSolaris taxonomy (http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/#Advice)
Interface have two main attributes: *Scope* and *Stability*
* *Scope* - _denotes the potential customers of the interface_.
For example many interfaces are merely internal or private interfaces of the implementation while others are public or external interfaces that applications or clients are expected to use. In posix, libc is an is an external or public interface, while large parts of the kernel are internal or private interfaces. In addition, some interfaces are targeted to some specific other subsystems. Identifying the scope helps define the customers or users of the interfaces and helps define the impact of breaking an interface. For example we may be willing to break the comaptibility of an interface whose scope is a small number of specific subsystems. One the other hand, one is unlikely to break a protocol interfaces that millions of internet users depend on.
The following are useful scopes in order of increasing/wider visibility
** *project-private*
*** the interface is for internal use _within_ the project and should not be used by applications. It is subject to change at anytime without notice. Most interfaces of a project are project private.
** *limited-private*
*** the interface is used by a specified set of projects or systems (typically closely related projects). Other projects or systems should not use the interface. Changes to the interface will be communicated/negotiated with the specified projects. For example, in the hadoop project, some interfaces are *hdfs-mapReduce-private* in that they are private to the hdfs and mapReduce projects.
** *company-private* (*_This not applicable to opensource projects such as Hadoop._* It is mentioned here for completeness.)
*** the interface can use used by other projects within a company.
** *public*
*** the interface is for general use by any application.
* *Stability* - _denotes when changes can be made to the interface that break compatibility_.
** *Stable*
*** Can evolve while retaining compatibility for minor release boundaries.; can break compatibility only at major release (ie. at m.0).
** *Evolving*
*** Evolving, but can break compatibility at minor release (i.e. m.x)
** *Unstable*
*** This usually makes sense for only private interfaces.
*** However one may call this out for a _supposedly_ public interface to highlight that it should not be used as an interface; for public interfaces, labeling it as *Not-an-interface* is probably more appropriate than "unstable".
**** Examples of publically visible interfaces that are unstable (ie not-an-interface): GUI, CLIs whose output format will change
** *Deprecated* - should not be used, will be removed in the future.
h2. FAQ
# What is the harm in applications using a private interface that is stable? How is it different than a public stable interface?
While a private interface marked as stable is targeted to change only at major releases, it may break at other times if the providers of that interface are willing to changes the internal users of that interface. Further, a public stable interface is less likely to break even at major releases (even though it is allowed to break compatibility) because the impact of the change is larger. *If you use a private interface (regardless of its stability) you run the risk of incompatibility*.
# Why bother declaring the stability of a private interface?
** To communicate the intent to its internal users.
** To provide guidelines to developers of the interface
** The stability may capture other internal properties of the system
*** e.g In HDFS, NN-DN protocol stability can help implement as rolling upgrades
*** e.g. In HDFS, FSImage stabilty can help provide more flexible roll backs.
# How will the classification be recorded for hadoop APIs?
** Each interface or class will have the scope and stability recorded using javadoc tags, annotation, or some other mechanim. What ever mechanism we choose, the classification must be visisble on the genrated java doc.
** APIs of private scope will not be part of the "public javadoc generated by ant (ie by the _ant target_ "javadoc"); they will only be generated for the developer javadoc (generated by _ant target_ "javadoc-dev")
** One can derive the scope of java classes and java interfaces by the scope of the package in which they are contained. Hence it is useful to declare the scope of each java package as public or private (along with the private scope variations).
h2. Proposed Classification for Hadoop Interfaces
* Scope Public
** Stable
*** FileSystem, MapReduce, Config, CLI (inlcuding output), parts of Mapred.lib, Job Logs API, instrumentation metrics. Audit logs
** Evolving
*** TFile, parts of Mapred.lib, some instrumentation metrics, jmx interface (till it becomes stable),
*** Job logs and job history ( Some tools, scripts and chukwa use this to analyze job processing)
** Not An interface
*** Web GUI
* Scope Private
** Limited-Private Evolving
*** RPC, Metrics (HDFS-MapReduce Private) - once stable, we can consider making these public-stable.
** Project-Private Stable
*** Intra-HDFS and MR protocols (facilitates rolling upgrades down the road)
*** FSImage
**** Note this will enable old versions of HDFS to read newer fsImage and hence enable more flexible roll backs.
**** Q. Should this be Project-Private Evolving instead?
**** Regardless of the stability of FSImage, new versions of HDFS have to be able to transparently convert older versions and provide roll-back.
** Project-Private Evolving
*** DFSClient (Q. should this be "project-private unstable"
** Project-Private Unstable
*** System logs
*** All implementation classes and interfaces not otherwise classified are considered to be project-private stable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665609#action_12665609 ]
Doug Cutting commented on HADOOP-5073:
--------------------------------------
> only things that are marked as stable are considered stable, everything else is inherently unstable/evolving
That's easier for developers to maintain, but I think more confusing for users, who might reasonably assume that documented features will be supported going forward unless marked otherwise. Documentation should be maintainable, but not at the expense of clear description. So I still prefer marking APIs that are unstable and leaving stable APIs as the unmarked case. We can use tools like jdiff to help us audit releases for new features that ought to be marked as unstable.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667491#action_12667491 ]
Jakob Homan commented on HADOOP-5073:
-------------------------------------
bq. If we cannot, then public declarations that appear in the documentation that are not explicitly noted as evolving must be considered stable, since that's what users might reasonably assume.
As long as we supply the annotation with a default value (such as stable) and ensure that each class and package that should have an annotation do have an annotation, this should happen pretty much automatically.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724070#action_12724070 ]
Steve Loughran commented on HADOOP-5073:
----------------------------------------
@Jiaqi
-that's a a separate problem, which is more logs ought to be designed for machine post-mortems, and hence marked as "please don't change". Which is something for a new bugrep. (and yes, I agree with you, stable log output is good)
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699977#action_12699977 ]
Jakob Homan commented on HADOOP-5073:
-------------------------------------
Had an offline talk with Sanjay and went over these issues.
One item that came up was the terminology. Scope may be a confusing term to use as it overloads a standard comp. sci term and may not be clear to new people approach our use of it. Since the goal of the scope classification is to indicate who should use the class, would audience be a more appropriate choice? Similarly with stability, it would probably be good to always refer to that as APIStability, lest our code gets tagged with "Unstable" all over and frightens away anyone looking it over to consider using Hadoop in production. Without proper context, stability and scope may not convey the appropriate meanings.
I've been playing around with using annotations approach and in general they work nicely. Annotations such as (rough versions without proper documentation):
{code}package annotations;
import java.lang.annotation.Documented;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@Documented
@Retention(RetentionPolicy.CLASS)
@Inherited
public @interface Audience {
public enum AUDIENCE {
/**
* the interface is for general use by any application.
*/
Public,
/**
* the interface is used by a specified set of projects or systems (typically closely related projects).
*/
LimitedPrivate,
/**
* Only for use within application, not for use by general developers
*/
ProjectPrivate }
AUDIENCE value();
}{code}
and
{code}package annotations;
import java.lang.annotation.Documented;
import java.lang.annotation.Inherited;
@Inherited
@Documented
public @interface APIStability {
public enum Stability { Stable, Evolving, Unstable };
Stability value();
}{code}
allows for reasonable, very clear documentation of code:
{code}package eccleston;
import static annotations.Audience.AUDIENCE.Public;
import annotations.APIStability;
import static annotations.APIStability.Stability.Evolving;
import static annotations.APIStability.Stability.Stable;
import annotations.Audience;
/**
* Time and Relative Dimensions in Space
*/
@Audience(Public)
@APIStability(Stable)
public class TARDIS {
public void materialize() { System.out.println("whoop! WHOop! WHOOP!"); }
public void dematerialize() { System.out.println("WHOOP! WHOop! whoop!"); }
@APIStability(Evolving) // We can mark individual methods as necessary
public void blendIn() { System.out.println("Now I'm a police box! That'll work."); }
}{code}
By using the @Documented meta-annotation, the audience and api-stability annotation appear in the javadoc. This is good:
{noformat}
eccleston
Class Tardis
java.lang.Object
eccleston.TARDIS
Direct Known Subclasses:
Type2TARDIS
@Audience(value=Public)
@APIStability(value=Stable)
public class TARDIS
extends java.lang.Object
Time and Relative Dimensions in Space
{noformat}
Unfortunately, there are two issues:
* Even though it's @Documented and @Inherited, the documentation doesn't get inherited, so even though subclasses of this type *do* have the @APIStability annotation (you can query for it using reflection), it doesn't show up in their javadoc. This makes it difficult to just mark a class as such-and-such stability or audience and assume that takes care of it down the line. This really seems wrong; certainly not the behavior I expected. I played with writing a doclet to do something like this, but the doclet interface is rather cheesy and doesn't allow you to modify the code information, just read it.
* Eclipse doesn't seem to show the annotations in its javadoc view, which makes it more difficult for coders who rely on these facilities for first-line identification of code and classes they should and shouldn't use.
I had hoped we could just use package-info.java to mark a class' audience and api-stability and have it propagate all the way down the line in the documentation, but that doesn't work. In addition to issue one above, the @Inherited meta-annotation can only be applied to classes. As a partial remedy to these issues, we could write a documentation tool to check if new code is using any unstable/private apis and emit a warning, as part of test-patch or such.
Taglets show up in the javadoc quite boldly, but I can't find a way to propagate them; we'd have to explicitly mark each section we wanted to tag (or write a tool to do it).
Since we can't just mark a top level class and have it show up all down the line, having default values may not be a good idea after all. It's probably best to explicitly set the audience/api-stability value when the annotation is applied.
So, the annotation approach may work but it's not as in-your-face as I was hoping for helping coders to immediately see which resources they should or shouldn't use in their development environments.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664692#action_12664692 ]
Doug Cutting commented on HADOOP-5073:
--------------------------------------
This sounds fine.
The implication on documentation should be something like:
- Public/stable should be documented.
- Public/evolving should documented, but must be well-marked as evolving.
- Private, limited or otherwise, should not be included in public documentation, but only in internal documents.
The marking for evolving could be as simple as adding "Evolving:" to the beginning of javadoc strings. Or perhaps we can add an annotation and doclet which adds "Evolving" in *bold*, as is done for deprecations. If evolving and/or deprecated features have Forrest documentation, then that too should include a marking.
We'd also need to add end-user documentation of this policy, including exactly what "Evolving" and "Deprecated" mean. This should not include any discussion of the private classifications, only our compatibility promises for public stuff, and that anything undocumented has no compatibility promises.
Does this make sense?
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667474#action_12667474 ]
Doug Cutting commented on HADOOP-5073:
--------------------------------------
> Annotations have the advantage that they can be included in the javadoc, providing a resource for developers.
Annotations and/or taglets may be a great tools to help implement this. Perhaps we can limit generated documentation to only include things whose annotated scope is public. If we cannot, then public declarations that appear in the documentation that are not explicitly noted as evolving must be considered stable, since that's what users might reasonably assume.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Jiaqi Tan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724073#action_12724073 ]
Jiaqi Tan commented on HADOOP-5073:
-----------------------------------
@Steve
I actually commented about this issue of logs here because logs are mentioned here as part of the interface definition, so I was hoping to get "debugging+profiling log statements" marked as part of the public interface that will eventually become stable. Sure, the actual statements to be marked should be part of a separate JIRA, but I wanted to get this defined in the interface as well.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665103#action_12665103 ]
Steve Loughran commented on HADOOP-5073:
----------------------------------------
I am +1 in the different scopes, and in favour of using Java 5 annotations to mark this up, rather than relying purely on package scoping to hide things. In the open source world, there's nothing to stop anyone putting their classes in your package to get at internal stuff, leaving you to field the support calls with your product name in all the stack traces.
I'd suggest that we have an @Stable marker to all interfaces that are considered stable; only things that are marked as stable are considered stable, everything else is inherently unstable/evolving
One thing I would change, the Web GUI is an interface. if you can access it programmatically -as fsck does- then it has stability issues too.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667456#action_12667456 ]
Jakob Homan commented on HADOOP-5073:
-------------------------------------
Annotations have the advantage that they can be included in the javadoc, providing a resource for developers.
To include an annotation in the javadoc, annotate the annotation with @documented and a runtime retention policy (this policy is required):
{noformat}
// Example annotation that is included in javadoc
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface PublicScope {
public enum Scope {STABLE, EVOLVING};
Scope scope() default Scope.STABLE;
}
{noformat}
and then annotate the class normally:
{noformat}
@PublicScope(scope=PublicScope.Scope.EVOLVING)
public class Tardis {
private int foo;
{noformat}
The generated Javadoc then includes the annotation, right below the direct known subclasses, and is hyperlinked to the annotation definition, which can include clarification of what the different values mean.
The actual annotating is pretty wordy (and my example wording is poor), so it'll be good to pick out a reasonable naming system.
Another nice Annotation feature is the ability to specify its target: package, class, field, etc.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5073) Hadoop 1.0 Interface Classification -
scope (visibility - public/private) and stability
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-5073:
---------------------------------
Description:
This jira proposes an interface classification for hadoop interfaces.
The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
was:
This jira proposes an interface classification for hadoop interfaces.
The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
h2. Interface Taxonomy - Scope & Stability Classification
The interface taxonomy classification provided here is for guidance to developers and users of interfaces.
The classification guides a developer to declare the scope (or targeted audience or users) of an interface and also its stability.
* *Benefits to the user of an interface*: Knows which interfaces to use or not use and their stability.
* *Benefits to the developer*: to prevent accidental changes of interfaces and hence accidental impact on users or other components or system. This is particularly useful in large systems with many developers who may not all have a shared state/history of the project.
This classification was derived from a taxonomy used inside Yahoo and
from the OpenSolaris taxonomy (http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/#Advice)
Interface have two main attributes: *Scope* and *Stability*
* *Scope* - _denotes the potential customers of the interface_.
For example many interfaces are merely internal or private interfaces of the implementation while others are public or external interfaces that applications or clients are expected to use. In posix, libc is an is an external or public interface, while large parts of the kernel are internal or private interfaces. In addition, some interfaces are targeted to some specific other subsystems. Identifying the scope helps define the customers or users of the interfaces and helps define the impact of breaking an interface. For example we may be willing to break the comaptibility of an interface whose scope is a small number of specific subsystems. One the other hand, one is unlikely to break a protocol interfaces that millions of internet users depend on.
The following are useful scopes in order of increasing/wider visibility
** *project-private*
*** the interface is for internal use _within_ the project and should not be used by applications. It is subject to change at anytime without notice. Most interfaces of a project are project private.
** *limited-private*
*** the interface is used by a specified set of projects or systems (typically closely related projects). Other projects or systems should not use the interface. Changes to the interface will be communicated/negotiated with the specified projects. For example, in the hadoop project, some interfaces are *hdfs-mapReduce-private* in that they are private to the hdfs and mapReduce projects.
** *company-private* (*_This not applicable to opensource projects such as Hadoop._* It is mentioned here for completeness.)
*** the interface can use used by other projects within a company.
** *public*
*** the interface is for general use by any application.
* *Stability* - _denotes when changes can be made to the interface that break compatibility_.
** *Stable*
*** Can evolve while retaining compatibility for minor release boundaries.; can break compatibility only at major release (ie. at m.0).
** *Evolving*
*** Evolving, but can break compatibility at minor release (i.e. m.x)
** *Unstable*
*** This usually makes sense for only private interfaces.
*** However one may call this out for a _supposedly_ public interface to highlight that it should not be used as an interface; for public interfaces, labeling it as *Not-an-interface* is probably more appropriate than "unstable".
**** Examples of publically visible interfaces that are unstable (ie not-an-interface): GUI, CLIs whose output format will change
** *Deprecated* - should not be used, will be removed in the future.
h2. FAQ
# What is the harm in applications using a private interface that is stable? How is it different than a public stable interface?
While a private interface marked as stable is targeted to change only at major releases, it may break at other times if the providers of that interface are willing to changes the internal users of that interface. Further, a public stable interface is less likely to break even at major releases (even though it is allowed to break compatibility) because the impact of the change is larger. *If you use a private interface (regardless of its stability) you run the risk of incompatibility*.
# Why bother declaring the stability of a private interface?
** To communicate the intent to its internal users.
** To provide guidelines to developers of the interface
** The stability may capture other internal properties of the system
*** e.g In HDFS, NN-DN protocol stability can help implement as rolling upgrades
*** e.g. In HDFS, FSImage stabilty can help provide more flexible roll backs.
# How will the classification be recorded for hadoop APIs?
** Each interface or class will have the scope and stability recorded using javadoc tags, annotation, or some other mechanim. What ever mechanism we choose, the classification must be visisble on the genrated java doc.
** APIs of private scope will not be part of the "public javadoc generated by ant (ie by the _ant target_ "javadoc"); they will only be generated for the developer javadoc (generated by _ant target_ "javadoc-dev")
** One can derive the scope of java classes and java interfaces by the scope of the package in which they are contained. Hence it is useful to declare the scope of each java package as public or private (along with the private scope variations).
h2. Proposed Classification for Hadoop Interfaces
* Scope Public
** Stable
*** FileSystem, MapReduce, Config, CLI (inlcuding output), parts of Mapred.lib, Job Logs API, instrumentation metrics. Audit logs
** Evolving
*** TFile, parts of Mapred.lib, some instrumentation metrics, jmx interface (till it becomes stable),
*** Job logs and job history ( Some tools, scripts and chukwa use this to analyze job processing)
** Not An interface
*** Web GUI
* Scope Private
** Limited-Private Evolving
*** RPC, Metrics (HDFS-MapReduce Private) - once stable, we can consider making these public-stable.
** Project-Private Stable
*** Intra-HDFS and MR protocols (facilitates rolling upgrades down the road)
*** FSImage
**** Note this will enable old versions of HDFS to read newer fsImage and hence enable more flexible roll backs.
**** Q. Should this be Project-Private Evolving instead?
**** Regardless of the stability of FSImage, new versions of HDFS have to be able to transparently convert older versions and provide roll-back.
** Project-Private Evolving
*** DFSClient (Q. should this be "project-private unstable"
** Project-Private Unstable
*** System logs
*** All implementation classes and interfaces not otherwise classified are considered to be project-private stable.
h2. Interface Taxonomy - Scope & Stability Classification
The interface taxonomy classification provided here is for guidance to developers and users of interfaces.
The classification guides a developer to declare the scope (or targeted audience or users) of an interface and also its stability.
* *Benefits to the user of an interface*: Knows which interfaces to use or not use and their stability.
* *Benefits to the developer*: to prevent accidental changes of interfaces and hence accidental impact on users or other components or system. This is particularly useful in large systems with many developers who may not all have a shared state/history of the project.
This classification was derived from a taxonomy used inside Yahoo and
from the OpenSolaris taxonomy (http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/#Advice)
Interface have two main attributes: *Scope* and *Stability*
* *Scope* - _denotes the potential customers of the interface_.
For example many interfaces are merely internal or private interfaces of the implementation while others are public or external interfaces that applications or clients are expected to use. In posix, libc is an is an external or public interface, while large parts of the kernel are internal or private interfaces. In addition, some interfaces are targeted to some specific other subsystems. Identifying the scope helps define the customers or users of the interfaces and helps define the impact of breaking an interface. For example we may be willing to break the comaptibility of an interface whose scope is a small number of specific subsystems. One the other hand, one is unlikely to break a protocol interfaces that millions of internet users depend on.
The following are useful scopes in order of increasing/wider visibility
** *project-private*
*** the interface is for internal use _within_ the project and should not be used by applications. It is subject to change at anytime without notice. Most interfaces of a project are project private.
** *limited-private*
*** the interface is used by a specified set of projects or systems (typically closely related projects). Other projects or systems should not use the interface. Changes to the interface will be communicated/negotiated with the specified projects. For example, in the hadoop project, some interfaces are *hdfs-mapReduce-private* in that they are private to the hdfs and mapReduce projects.
** *company-private* (*_This not applicable to opensource projects such as Hadoop._* It is mentioned here for completeness.)
*** the interface can use used by other projects within a company.
** *public*
*** the interface is for general use by any application.
* *Stability* - _denotes when changes can be made to the interface that break compatibility_.
** *Stable*
*** Can evolve while retaining compatibility for minor release boundaries.; can break compatibility only at major release (ie. at m.0).
** *Evolving*
*** Evolving, but can break compatibility at minor release (i.e. m.x)
** *Unstable*
*** This usually makes sense for only private interfaces.
*** However one may call this out for a _supposedly_ public interface to highlight that it should not be used as an interface; for public interfaces, labeling it as *Not-an-interface* is probably more appropriate than "unstable".
**** Examples of publically visible interfaces that are unstable (ie not-an-interface): GUI, CLIs whose output format will change
** *Deprecated* - should not be used, will be removed in the future.
h2. FAQ
# What is the harm in applications using a private interface that is stable? How is it different than a public stable interface?
While a private interface marked as stable is targeted to change only at major releases, it may break at other times if the providers of that interface are willing to changes the internal users of that interface. Further, a public stable interface is less likely to break even at major releases (even though it is allowed to break compatibility) because the impact of the change is larger. *If you use a private interface (regardless of its stability) you run the risk of incompatibility*.
# Why bother declaring the stability of a private interface?
** To communicate the intent to its internal users.
** To provide guidelines to developers of the interface
** The stability may capture other internal properties of the system
*** e.g In HDFS, NN-DN protocol stability can help implement as rolling upgrades
*** e.g. In HDFS, FSImage stabilty can help provide more flexible roll backs.
# How will the classification be recorded for hadoop APIs?
** Each interface or class will have the scope and stability recorded using javadoc tags, annotation, or some other mechanim. What ever mechanism we choose, the classification must be visisble on the genrated java doc.
** APIs of private scope will not be part of the "public javadoc generated by ant (ie by the _ant target_ "javadoc"); they will only be generated for the developer javadoc (generated by _ant target_ "javadoc-dev")
** One can derive the scope of java classes and java interfaces by the scope of the package in which they are contained. Hence it is useful to declare the scope of each java package as public or private (along with the private scope variations).
h2. Proposed Classification for Hadoop Interfaces
* Scope Public
** Stable
*** FileSystem, MapReduce, Config, CLI (inlcuding output), parts of Mapred.lib, Job Logs API, instrumentation metrics. Audit logs
** Evolving
*** TFile, parts of Mapred.lib, some instrumentation metrics, jmx interface (till it becomes stable),
*** Job logs and job history ( Some tools, scripts and chukwa use this to analyze job processing)
** Not An interface
*** Web GUI
* Scope Private
** Limited-Private Evolving
*** RPC, Metrics (HDFS-MapReduce Private) - once stable, we can consider making these public-stable.
** Project-Private Stable
*** Intra-HDFS and MR protocols (facilitates rolling upgrades down the road)
*** FSImage
**** Note this will enable old versions of HDFS to read newer fsImage and hence enable more flexible roll backs.
**** Q. Should this be Project-Private Evolving instead?
**** Regardless of the stability of FSImage, new versions of HDFS have to be able to transparently convert older versions and provide roll-back.
** Project-Private Evolving
*** DFSClient (Q. should this be "project-private unstable"
** Project-Private Unstable
*** System logs
*** All implementation classes and interfaces not otherwise classified are considered to be project-private stable.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-5073) Hadoop 1.0 Interface
Classification - scope (visibility - public/private) and stability
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664679#action_12664679 ]
sanjay.radia edited comment on HADOOP-5073 at 1/16/09 1:12 PM:
---------------------------------------------------------------
A proposal:
h2. Interface Taxonomy - Scope & Stability Classification
The interface taxonomy classification provided here is for guidance to developers and users of interfaces.
The classification guides a developer to declare the scope (or targeted audience or users) of an interface and also its stability.
* *Benefits to the user of an interface*: Knows which interfaces to use or not use and their stability.
* *Benefits to the developer*: to prevent accidental changes of interfaces and hence accidental impact on users or other components or system. This is particularly useful in large systems with many developers who may not all have a shared state/history of the project.
This classification was derived from a taxonomy used inside Yahoo and
from the OpenSolaris taxonomy (http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/#Advice)
Interface have two main attributes: *Scope* and *Stability*
* *Scope* - _denotes the potential customers of the interface_.
For example many interfaces are merely internal or private interfaces of the implementation while others are public or external interfaces that applications or clients are expected to use. In posix, libc is an is an external or public interface, while large parts of the kernel are internal or private interfaces. In addition, some interfaces are targeted to some specific other subsystems. Identifying the scope helps define the customers or users of the interfaces and helps define the impact of breaking an interface. For example we may be willing to break the comaptibility of an interface whose scope is a small number of specific subsystems. One the other hand, one is unlikely to break a protocol interfaces that millions of internet users depend on.
The following are useful scopes in order of increasing/wider visibility
** *project-private*
*** the interface is for internal use _within_ the project and should not be used by applications. It is subject to change at anytime without notice. Most interfaces of a project are project private.
** *limited-private*
*** the interface is used by a specified set of projects or systems (typically closely related projects). Other projects or systems should not use the interface. Changes to the interface will be communicated/negotiated with the specified projects. For example, in the hadoop project, some interfaces are *hdfs-mapReduce-private* in that they are private to the hdfs and mapReduce projects.
** *company-private* (*_This not applicable to opensource projects such as Hadoop._* It is mentioned here for completeness.)
*** the interface can use used by other projects within a company.
** *public*
*** the interface is for general use by any application.
* *Stability* - _denotes when changes can be made to the interface that break compatibility_.
** *Stable*
*** Can evolve while retaining compatibility for minor release boundaries.; can break compatibility only at major release (ie. at m.0).
** *Evolving*
*** Evolving, but can break compatibility at minor release (i.e. m.x)
** *Unstable*
*** This usually makes sense for only private interfaces.
*** However one may call this out for a _supposedly_ public interface to highlight that it should not be used as an interface; for public interfaces, labeling it as *Not-an-interface* is probably more appropriate than "unstable".
**** Examples of publically visible interfaces that are unstable (ie not-an-interface): GUI, CLIs whose output format will change
** *Deprecated* - should not be used, will be removed in the future.
h2. FAQ
# What is the harm in applications using a private interface that is stable? How is it different than a public stable interface?
While a private interface marked as stable is targeted to change only at major releases, it may break at other times if the providers of that interface are willing to changes the internal users of that interface. Further, a public stable interface is less likely to break even at major releases (even though it is allowed to break compatibility) because the impact of the change is larger. *If you use a private interface (regardless of its stability) you run the risk of incompatibility*.
# Why bother declaring the stability of a private interface?
** To communicate the intent to its internal users.
** To provide guidelines to developers of the interface
** The stability may capture other internal properties of the system
*** e.g In HDFS, NN-DN protocol stability can help implement as rolling upgrades
*** e.g. In HDFS, FSImage stabilty can help provide more flexible roll backs.
# How will the classification be recorded for hadoop APIs?
** Each interface or class will have the scope and stability recorded using javadoc tags, annotation, or some other mechanim. What ever mechanism we choose, the classification must be visisble on the genrated java doc.
** APIs of private scope will not be part of the "public javadoc generated by ant (ie by the _ant target_ "javadoc"); they will only be generated for the developer javadoc (generated by _ant target_ "javadoc-dev")
** One can derive the scope of java classes and java interfaces by the scope of the package in which they are contained. Hence it is useful to declare the scope of each java package as public or private (along with the private scope variations).
h2. Proposed Classification for Hadoop Interfaces
* Scope Public
** Stable
*** FileSystem, MapReduce, Config, CLI (inlcuding output), parts of Mapred.lib, Job Logs API, instrumentation metrics. Audit logs
** Evolving
*** TFile, parts of Mapred.lib, some instrumentation metrics, jmx interface (till it becomes stable),
*** Job logs and job history ( Some tools, scripts and chukwa use this to analyze job processing)
** Not An interface
*** Web GUI
* Scope Private
** Limited-Private Evolving
*** RPC, Metrics (HDFS-MapReduce Private) - once stable, we can consider making these public-stable.
** Project-Private Stable
*** Intra-HDFS and MR protocols (facilitates rolling upgrades down the road)
*** FSImage
**** Note this will enable old versions of HDFS to read newer fsImage and hence enable more flexible roll backs.
**** Q. Should this be Project-Private Evolving instead?
**** Regardless of the stability of FSImage, new versions of HDFS have to be able to transparently convert older versions and provide roll-back.
** Project-Private Evolving
*** DFSClient (Q. should this be "project-private unstable"
** Project-Private Unstable
*** System logs
*** All implementation classes and interfaces not otherwise classified are considered to be project-private stable.
was (Author: sanjay.radia):
h2. Interface Taxonomy - Scope & Stability Classification
The interface taxonomy classification provided here is for guidance to developers and users of interfaces.
The classification guides a developer to declare the scope (or targeted audience or users) of an interface and also its stability.
* *Benefits to the user of an interface*: Knows which interfaces to use or not use and their stability.
* *Benefits to the developer*: to prevent accidental changes of interfaces and hence accidental impact on users or other components or system. This is particularly useful in large systems with many developers who may not all have a shared state/history of the project.
This classification was derived from a taxonomy used inside Yahoo and
from the OpenSolaris taxonomy (http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/#Advice)
Interface have two main attributes: *Scope* and *Stability*
* *Scope* - _denotes the potential customers of the interface_.
For example many interfaces are merely internal or private interfaces of the implementation while others are public or external interfaces that applications or clients are expected to use. In posix, libc is an is an external or public interface, while large parts of the kernel are internal or private interfaces. In addition, some interfaces are targeted to some specific other subsystems. Identifying the scope helps define the customers or users of the interfaces and helps define the impact of breaking an interface. For example we may be willing to break the comaptibility of an interface whose scope is a small number of specific subsystems. One the other hand, one is unlikely to break a protocol interfaces that millions of internet users depend on.
The following are useful scopes in order of increasing/wider visibility
** *project-private*
*** the interface is for internal use _within_ the project and should not be used by applications. It is subject to change at anytime without notice. Most interfaces of a project are project private.
** *limited-private*
*** the interface is used by a specified set of projects or systems (typically closely related projects). Other projects or systems should not use the interface. Changes to the interface will be communicated/negotiated with the specified projects. For example, in the hadoop project, some interfaces are *hdfs-mapReduce-private* in that they are private to the hdfs and mapReduce projects.
** *company-private* (*_This not applicable to opensource projects such as Hadoop._* It is mentioned here for completeness.)
*** the interface can use used by other projects within a company.
** *public*
*** the interface is for general use by any application.
* *Stability* - _denotes when changes can be made to the interface that break compatibility_.
** *Stable*
*** Can evolve while retaining compatibility for minor release boundaries.; can break compatibility only at major release (ie. at m.0).
** *Evolving*
*** Evolving, but can break compatibility at minor release (i.e. m.x)
** *Unstable*
*** This usually makes sense for only private interfaces.
*** However one may call this out for a _supposedly_ public interface to highlight that it should not be used as an interface; for public interfaces, labeling it as *Not-an-interface* is probably more appropriate than "unstable".
**** Examples of publically visible interfaces that are unstable (ie not-an-interface): GUI, CLIs whose output format will change
** *Deprecated* - should not be used, will be removed in the future.
h2. FAQ
# What is the harm in applications using a private interface that is stable? How is it different than a public stable interface?
While a private interface marked as stable is targeted to change only at major releases, it may break at other times if the providers of that interface are willing to changes the internal users of that interface. Further, a public stable interface is less likely to break even at major releases (even though it is allowed to break compatibility) because the impact of the change is larger. *If you use a private interface (regardless of its stability) you run the risk of incompatibility*.
# Why bother declaring the stability of a private interface?
** To communicate the intent to its internal users.
** To provide guidelines to developers of the interface
** The stability may capture other internal properties of the system
*** e.g In HDFS, NN-DN protocol stability can help implement as rolling upgrades
*** e.g. In HDFS, FSImage stabilty can help provide more flexible roll backs.
# How will the classification be recorded for hadoop APIs?
** Each interface or class will have the scope and stability recorded using javadoc tags, annotation, or some other mechanim. What ever mechanism we choose, the classification must be visisble on the genrated java doc.
** APIs of private scope will not be part of the "public javadoc generated by ant (ie by the _ant target_ "javadoc"); they will only be generated for the developer javadoc (generated by _ant target_ "javadoc-dev")
** One can derive the scope of java classes and java interfaces by the scope of the package in which they are contained. Hence it is useful to declare the scope of each java package as public or private (along with the private scope variations).
h2. Proposed Classification for Hadoop Interfaces
* Scope Public
** Stable
*** FileSystem, MapReduce, Config, CLI (inlcuding output), parts of Mapred.lib, Job Logs API, instrumentation metrics. Audit logs
** Evolving
*** TFile, parts of Mapred.lib, some instrumentation metrics, jmx interface (till it becomes stable),
*** Job logs and job history ( Some tools, scripts and chukwa use this to analyze job processing)
** Not An interface
*** Web GUI
* Scope Private
** Limited-Private Evolving
*** RPC, Metrics (HDFS-MapReduce Private) - once stable, we can consider making these public-stable.
** Project-Private Stable
*** Intra-HDFS and MR protocols (facilitates rolling upgrades down the road)
*** FSImage
**** Note this will enable old versions of HDFS to read newer fsImage and hence enable more flexible roll backs.
**** Q. Should this be Project-Private Evolving instead?
**** Regardless of the stability of FSImage, new versions of HDFS have to be able to transparently convert older versions and provide roll-back.
** Project-Private Evolving
*** DFSClient (Q. should this be "project-private unstable"
** Project-Private Unstable
*** System logs
*** All implementation classes and interfaces not otherwise classified are considered to be project-private stable.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Core
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
- scope (visibility - public/private) and stability
Posted by "Jiaqi Tan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723738#action_12723738 ]
Jiaqi Tan commented on HADOOP-5073:
-----------------------------------
I would like to propose that a particular subset of logging statements currently in Hadoop be made Public-Evolving, and eventually Public-Stable, for the purpose of lightweight tracing for studying Hadoop behavior. This has various applications, such as for diagnosing failures and visualizing behavior, and using log statements that are already being currently generated gives us a nice, low-overhead way for doing this.
We have been using Hadoop logs for diagnosis for some time now, and while the log statements have been relatively stable, we have had to make minor parser modifications between Hadoop versions, and there is currently no guarantee that the statements we find useful will be here to stay. It would be nice if we can articulate some of the existing log statements that help us in diagnosis, and if those statements can be made part of a public interface. Then, parsing and analysis tools can be produced based on this stable interface.
This also relates to Chukwa, as far as us implementing our log-based analysis and diagnosis tools as part of the Chukwa project.
> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-5073
> URL: https://issues.apache.org/jira/browse/HADOOP-5073
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.