htrace-core compatibility policy for htrace 4.x

[Colin McCabe <cm...@alumni.cmu.edu>]

Hi all,

In the recent 4.0 release, we changed the htrace-core API. The API
that programs use to create traces, annotations, etc. (aka the "Java
client API") went through some changes. This was necessary to clean up
some core architectural issues (such as the use of overly short 64 bit
IDs that will collide in a real-world deployment, or the overuse of
globals.)

Since we want to make it easy for projects to integrate with HTrace, I
think we should have some compatibility rules for htrace-core for the
future.

Specifically, I think that we should include only backwards-compatible
changes to the htrace-core API in HTrace 4.x So, for example, adding a
new function is OK. Deleting an existing function or altering it in an
incompatible way is not. It is OK to add a new function to a public
abstract base class (provided you also add a default implementation in
the base), but not to add a new function to a public interface, since
that would break compilation.

We should save incompatible changes for HTrace 5.x. In general, each
"major release" such as 4.x or 5.x should contain only compatible
changes to htrace-core. There should be no guarantees between 4.x and
5.x, or between any major releases-- this is the time to address
architectural debt that can't be resolved any other way. Major
releases should change the namespace of htrace-core classes so that
both a 4.x and a 5.x jar can reside on the same CLASSPATH, similar to
how we did with 3.x and 4.x. This is important because it will require
some time for downstream projects to upgrade from 4.x to 5.x, and in
the meantime we must avoid CLASSPATH conflict issues. There is no
requirement that tracing work when the major version of the client and
the span receivers are different. However, the programs themselves
should function.

Let's focus on just compatibility rules for htrace-core right now,
since that's where our integration issues are. The other subprojects
of htrace generally don't have the same integration issues. For
example, it is easy for an admin to standardize on a single version of
htrace-hbase or htrace-htraced across the entire cluster. They simply
install the jars for the version they want. It is not easy for that
same admin to standardize htrace-core, since they might have Hadoop
pulling in 4.1 and HBase pulling in 4.0. The different subprojects are
also at different levels of maturity. For example, htrace-flume is
still very immature, whereas htrace-htraced is starting to get more
mature. So I think the subprojects should come up with their own
compatibility policies rather than trying to be one-size-fits-all.

This policy should only apply to publicly visible symbols in
htrace-core, not to private or package-private symbols.  Test
functions should also not be covered, since they don't appear in the
final htrace-core jar.

I think having a compatibility policy for htrace-core will be very
nice for the users of our core APIs. Let me know what you think.

best,
Colin

Re: htrace-core compatibility policy for htrace 4.x

[Nick Dimiduk <nd...@gmail.com>]

I think this is the right direction. We'll need a stable API to have any
hope of adoption outside of the Hadoop distro vendors' "walled garden" of a
given release. This is the nature of such a project designed to integrate
throughout a complex stack.

On Monday, September 14, 2015, Colin McCabe <cm...@alumni.cmu.edu> wrote:

> Hi all,
>
> In the recent 4.0 release, we changed the htrace-core API. The API
> that programs use to create traces, annotations, etc. (aka the "Java
> client API") went through some changes. This was necessary to clean up
> some core architectural issues (such as the use of overly short 64 bit
> IDs that will collide in a real-world deployment, or the overuse of
> globals.)
>
> Since we want to make it easy for projects to integrate with HTrace, I
> think we should have some compatibility rules for htrace-core for the
> future.
>
> Specifically, I think that we should include only backwards-compatible
> changes to the htrace-core API in HTrace 4.x So, for example, adding a
> new function is OK. Deleting an existing function or altering it in an
> incompatible way is not. It is OK to add a new function to a public
> abstract base class (provided you also add a default implementation in
> the base), but not to add a new function to a public interface, since
> that would break compilation.
>
> We should save incompatible changes for HTrace 5.x. In general, each
> "major release" such as 4.x or 5.x should contain only compatible
> changes to htrace-core. There should be no guarantees between 4.x and
> 5.x, or between any major releases-- this is the time to address
> architectural debt that can't be resolved any other way. Major
> releases should change the namespace of htrace-core classes so that
> both a 4.x and a 5.x jar can reside on the same CLASSPATH, similar to
> how we did with 3.x and 4.x. This is important because it will require
> some time for downstream projects to upgrade from 4.x to 5.x, and in
> the meantime we must avoid CLASSPATH conflict issues. There is no
> requirement that tracing work when the major version of the client and
> the span receivers are different. However, the programs themselves
> should function.
>
> Let's focus on just compatibility rules for htrace-core right now,
> since that's where our integration issues are. The other subprojects
> of htrace generally don't have the same integration issues. For
> example, it is easy for an admin to standardize on a single version of
> htrace-hbase or htrace-htraced across the entire cluster. They simply
> install the jars for the version they want. It is not easy for that
> same admin to standardize htrace-core, since they might have Hadoop
> pulling in 4.1 and HBase pulling in 4.0. The different subprojects are
> also at different levels of maturity. For example, htrace-flume is
> still very immature, whereas htrace-htraced is starting to get more
> mature. So I think the subprojects should come up with their own
> compatibility policies rather than trying to be one-size-fits-all.
>
> This policy should only apply to publicly visible symbols in
> htrace-core, not to private or package-private symbols.  Test
> functions should also not be covered, since they don't appear in the
> final htrace-core jar.
>
> I think having a compatibility policy for htrace-core will be very
> nice for the users of our core APIs. Let me know what you think.
>
> best,
> Colin
>

Re: htrace-core compatibility policy for htrace 4.x

[Masatake Iwasaki <iw...@oss.nttdata.co.jp>]

I agree with the policy.


 > Major releases should change the namespace of htrace-core classes so that
 > both a 4.x and a 5.x jar can reside on the same CLASSPATH

Should we have a namespace convention for this?
We moved classes from org.apache.htrace to org.apache.htrace.core on 4.0
but need new word other than "core" for 5.0.
Simple way for this would be containing version number in package name
like "org.apache.htrace.5".


 > Let's focus on just compatibility rules for htrace-core right now,
 > since that's where our integration issues are. The other subprojects
 > of htrace generally don't have the same integration issues.

I thinks it is reasonable.
Any version of receiver with the same major version keeps working
as far as htrace-core keeps compatibility.


Thanks,
Masatake Iwasaki


On 9/15/15 09:13, Colin McCabe wrote:
> Hi all,
>
> In the recent 4.0 release, we changed the htrace-core API. The API
> that programs use to create traces, annotations, etc. (aka the "Java
> client API") went through some changes. This was necessary to clean up
> some core architectural issues (such as the use of overly short 64 bit
> IDs that will collide in a real-world deployment, or the overuse of
> globals.)
>
> Since we want to make it easy for projects to integrate with HTrace, I
> think we should have some compatibility rules for htrace-core for the
> future.
>
> Specifically, I think that we should include only backwards-compatible
> changes to the htrace-core API in HTrace 4.x So, for example, adding a
> new function is OK. Deleting an existing function or altering it in an
> incompatible way is not. It is OK to add a new function to a public
> abstract base class (provided you also add a default implementation in
> the base), but not to add a new function to a public interface, since
> that would break compilation.
>
> We should save incompatible changes for HTrace 5.x. In general, each
> "major release" such as 4.x or 5.x should contain only compatible
> changes to htrace-core. There should be no guarantees between 4.x and
> 5.x, or between any major releases-- this is the time to address
> architectural debt that can't be resolved any other way. Major
> releases should change the namespace of htrace-core classes so that
> both a 4.x and a 5.x jar can reside on the same CLASSPATH, similar to
> how we did with 3.x and 4.x. This is important because it will require
> some time for downstream projects to upgrade from 4.x to 5.x, and in
> the meantime we must avoid CLASSPATH conflict issues. There is no
> requirement that tracing work when the major version of the client and
> the span receivers are different. However, the programs themselves
> should function.
>
> Let's focus on just compatibility rules for htrace-core right now,
> since that's where our integration issues are. The other subprojects
> of htrace generally don't have the same integration issues. For
> example, it is easy for an admin to standardize on a single version of
> htrace-hbase or htrace-htraced across the entire cluster. They simply
> install the jars for the version they want. It is not easy for that
> same admin to standardize htrace-core, since they might have Hadoop
> pulling in 4.1 and HBase pulling in 4.0. The different subprojects are
> also at different levels of maturity. For example, htrace-flume is
> still very immature, whereas htrace-htraced is starting to get more
> mature. So I think the subprojects should come up with their own
> compatibility policies rather than trying to be one-size-fits-all.
>
> This policy should only apply to publicly visible symbols in
> htrace-core, not to private or package-private symbols.  Test
> functions should also not be covered, since they don't appear in the
> final htrace-core jar.
>
> I think having a compatibility policy for htrace-core will be very
> nice for the users of our core APIs. Let me know what you think.
>
> best,
> Colin

Re: htrace-core compatibility policy for htrace 4.x

["Colin P. McCabe" <cm...@apache.org>]

On Tue, Sep 15, 2015 at 5:42 PM, Masatake Iwasaki
<iw...@oss.nttdata.co.jp> wrote:
> I agree with the policy.
>
>
>> Major releases should change the namespace of htrace-core classes so that
>> both a 4.x and a 5.x jar can reside on the same CLASSPATH
>
> Should we have a namespace convention for this?
> We moved classes from org.apache.htrace to org.apache.htrace.core on 4.0
> but need new word other than "core" for 5.0.
> Simple way for this would be containing version number in package name
> like "org.apache.htrace.5".

Hmm, interesting idea.  org.apache.htrace5 might be nice too.

We can probably wait a bit before deciding on the namespace name...
hopefully we won't need 5.x for a while :)

Colin

>
>
>> Let's focus on just compatibility rules for htrace-core right now,
>> since that's where our integration issues are. The other subprojects
>> of htrace generally don't have the same integration issues.
>
> I thinks it is reasonable.
> Any version of receiver with the same major version keeps working
> as far as htrace-core keeps compatibility.
>
>
> Thanks,
> Masatake Iwasaki
>
>
>
> On 9/15/15 09:13, Colin McCabe wrote:
>>
>> Hi all,
>>
>> In the recent 4.0 release, we changed the htrace-core API. The API
>> that programs use to create traces, annotations, etc. (aka the "Java
>> client API") went through some changes. This was necessary to clean up
>> some core architectural issues (such as the use of overly short 64 bit
>> IDs that will collide in a real-world deployment, or the overuse of
>> globals.)
>>
>> Since we want to make it easy for projects to integrate with HTrace, I
>> think we should have some compatibility rules for htrace-core for the
>> future.
>>
>> Specifically, I think that we should include only backwards-compatible
>> changes to the htrace-core API in HTrace 4.x So, for example, adding a
>> new function is OK. Deleting an existing function or altering it in an
>> incompatible way is not. It is OK to add a new function to a public
>> abstract base class (provided you also add a default implementation in
>> the base), but not to add a new function to a public interface, since
>> that would break compilation.
>>
>> We should save incompatible changes for HTrace 5.x. In general, each
>> "major release" such as 4.x or 5.x should contain only compatible
>> changes to htrace-core. There should be no guarantees between 4.x and
>> 5.x, or between any major releases-- this is the time to address
>> architectural debt that can't be resolved any other way. Major
>> releases should change the namespace of htrace-core classes so that
>> both a 4.x and a 5.x jar can reside on the same CLASSPATH, similar to
>> how we did with 3.x and 4.x. This is important because it will require
>> some time for downstream projects to upgrade from 4.x to 5.x, and in
>> the meantime we must avoid CLASSPATH conflict issues. There is no
>> requirement that tracing work when the major version of the client and
>> the span receivers are different. However, the programs themselves
>> should function.
>>
>> Let's focus on just compatibility rules for htrace-core right now,
>> since that's where our integration issues are. The other subprojects
>> of htrace generally don't have the same integration issues. For
>> example, it is easy for an admin to standardize on a single version of
>> htrace-hbase or htrace-htraced across the entire cluster. They simply
>> install the jars for the version they want. It is not easy for that
>> same admin to standardize htrace-core, since they might have Hadoop
>> pulling in 4.1 and HBase pulling in 4.0. The different subprojects are
>> also at different levels of maturity. For example, htrace-flume is
>> still very immature, whereas htrace-htraced is starting to get more
>> mature. So I think the subprojects should come up with their own
>> compatibility policies rather than trying to be one-size-fits-all.
>>
>> This policy should only apply to publicly visible symbols in
>> htrace-core, not to private or package-private symbols.  Test
>> functions should also not be covered, since they don't appear in the
>> final htrace-core jar.
>>
>> I think having a compatibility policy for htrace-core will be very
>> nice for the users of our core APIs. Let me know what you think.
>>
>> best,
>> Colin
>
>