MIMA Compatiblity Checks

[Patrick Wendell <pw...@gmail.com>]

Hey All,

Some people may have noticed PR failures due to binary compatibility
checks. We've had these enabled in several of the sub-modules since
the 0.9.0 release but we've turned them on in Spark core post 1.0.0
which has much higher churn.

The checks are based on the "migration manager" tool from Typesafe.
One issue is that tool doesn't support package-private declarations of
classes or methods. Prashant Sharma has built instrumentation that
adds partial support for package-privacy (via a workaround) but since
there isn't really native support for this in MIMA we are still
finding cases in which we trigger false positives.

In the next week or two we'll make it a priority to handle more of
these false-positive cases. In the mean time users can add manual
excludes to:

project/MimaExcludes.scala

to avoid triggering warnings for certain issues.

This is definitely annoying - sorry about that. Unfortunately we are
the first open source Scala project to ever do this, so we are dealing
with uncharted territory.

Longer term I'd actually like to see us just write our own sbt-based
tool to do this in a better way (we've had trouble trying to extend
MIMA itself, it e.g. has copy-pasted code in it from an old version of
the scala compiler). If someone in the community is a Scala fan and
wants to take that on, I'm happy to give more details.

- Patrick

Re: MIMA Compatiblity Checks

[Reynold Xin <rx...@databricks.com>]

You can take a look at

https://github.com/apache/spark/blob/master/dev/run-tests

dev/mima


On Thu, Jul 10, 2014 at 12:21 AM, Liu, Raymond <ra...@intel.com>
wrote:

> so how to run the check locally?
>
> On master tree, sbt mimaReportBinaryIssues Seems to lead to a lot of
> errors reported. Do we need to modify SparkBuilder.scala etc to run it
> locally? Could not figure out how Jekins run the check on its console
> outputs.
>
>
> Best Regards,
> Raymond Liu
>
> -----Original Message-----
> From: Patrick Wendell [mailto:pwendell@gmail.com]
> Sent: Monday, June 09, 2014 3:40 AM
> To: dev@spark.apache.org
> Subject: MIMA Compatiblity Checks
>
> Hey All,
>
> Some people may have noticed PR failures due to binary compatibility
> checks. We've had these enabled in several of the sub-modules since the
> 0.9.0 release but we've turned them on in Spark core post 1.0.0 which has
> much higher churn.
>
> The checks are based on the "migration manager" tool from Typesafe.
> One issue is that tool doesn't support package-private declarations of
> classes or methods. Prashant Sharma has built instrumentation that adds
> partial support for package-privacy (via a workaround) but since there
> isn't really native support for this in MIMA we are still finding cases in
> which we trigger false positives.
>
> In the next week or two we'll make it a priority to handle more of these
> false-positive cases. In the mean time users can add manual excludes to:
>
> project/MimaExcludes.scala
>
> to avoid triggering warnings for certain issues.
>
> This is definitely annoying - sorry about that. Unfortunately we are the
> first open source Scala project to ever do this, so we are dealing with
> uncharted territory.
>
> Longer term I'd actually like to see us just write our own sbt-based tool
> to do this in a better way (we've had trouble trying to extend MIMA itself,
> it e.g. has copy-pasted code in it from an old version of the scala
> compiler). If someone in the community is a Scala fan and wants to take
> that on, I'm happy to give more details.
>
> - Patrick
>

RE: MIMA Compatiblity Checks

["Liu, Raymond" <ra...@intel.com>]

so how to run the check locally?

On master tree, sbt mimaReportBinaryIssues Seems to lead to a lot of errors reported. Do we need to modify SparkBuilder.scala etc to run it locally? Could not figure out how Jekins run the check on its console outputs.


Best Regards,
Raymond Liu

-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com] 
Sent: Monday, June 09, 2014 3:40 AM
To: dev@spark.apache.org
Subject: MIMA Compatiblity Checks

Hey All,

Some people may have noticed PR failures due to binary compatibility checks. We've had these enabled in several of the sub-modules since the 0.9.0 release but we've turned them on in Spark core post 1.0.0 which has much higher churn.

The checks are based on the "migration manager" tool from Typesafe.
One issue is that tool doesn't support package-private declarations of classes or methods. Prashant Sharma has built instrumentation that adds partial support for package-privacy (via a workaround) but since there isn't really native support for this in MIMA we are still finding cases in which we trigger false positives.

In the next week or two we'll make it a priority to handle more of these false-positive cases. In the mean time users can add manual excludes to:

project/MimaExcludes.scala

to avoid triggering warnings for certain issues.

This is definitely annoying - sorry about that. Unfortunately we are the first open source Scala project to ever do this, so we are dealing with uncharted territory.

Longer term I'd actually like to see us just write our own sbt-based tool to do this in a better way (we've had trouble trying to extend MIMA itself, it e.g. has copy-pasted code in it from an old version of the scala compiler). If someone in the community is a Scala fan and wants to take that on, I'm happy to give more details.

- Patrick