You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2017/03/27 09:53:36 UTC
spark-website git commit: added section about mima
Repository: spark-website
Updated Branches:
refs/heads/asf-site 39838046c -> 8b27c470c
added section about mima
Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/8b27c470
Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/8b27c470
Diff: http://git-wip-us.apache.org/repos/asf/spark-website/diff/8b27c470
Branch: refs/heads/asf-site
Commit: 8b27c470c32b30f6bddafd4cdec2b75b5b975fb6
Parents: 3983804
Author: Benjamin Fradet <be...@gmail.com>
Authored: Sat Mar 25 21:48:40 2017 +0000
Committer: Benjamin Fradet <be...@gmail.com>
Committed: Sun Mar 26 18:14:45 2017 +0100
----------------------------------------------------------------------
developer-tools.md | 56 +++++++++++++++++++++++++++++++++++++++++-
site/developer-tools.html | 51 +++++++++++++++++++++++++++++++++++++-
2 files changed, 105 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark-website/blob/8b27c470/developer-tools.md
----------------------------------------------------------------------
diff --git a/developer-tools.md b/developer-tools.md
index 88f3f36..547d8aa 100644
--- a/developer-tools.md
+++ b/developer-tools.md
@@ -111,6 +111,60 @@ To run individual Java tests, you can use the `-Dtest` flag:
build/mvn test -DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite test
```
+<h3>Binary compatibility</h3>
+
+To ensure binary compatibility, Spark uses [MiMa](https://github.com/typesafehub/migration-manager).
+
+<h4>Ensuring binary compatibility</h4>
+
+When working on an issue, it's always a good idea to check that your changes do
+not introduce binary incompatibilities before opening a pull request.
+
+You can do so by running the following command:
+
+```
+$ dev/mima
+```
+
+A binary incompatibility reported by MiMa might look like the following:
+
+```
+[error] method this(org.apache.spark.sql.Dataset)Unit in class org.apache.spark.SomeClass does not have a correspondent in current version
+[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SomeClass.this")
+```
+
+If you open a pull request containing binary incompatibilities anyway, Jenkins
+will remind you by failing the test build with the following message:
+
+```
+Test build #xx has finished for PR yy at commit ffffff.
+
+ This patch fails MiMa tests.
+ This patch merges cleanly.
+ This patch adds no public classes.
+```
+
+<h4>Solving a binary incompatibility</h4>
+
+If you believe that your binary incompatibilies are justified or that MiMa
+reported false positives (e.g. the reported binary incompatibilities are about a
+non-user facing API), you can filter them out by adding an exclusion in
+[project/MimaExcludes.scala](https://github.com/apache/spark/blob/master/project/MimaExcludes.scala)
+containing what was suggested by the MiMa report and a comment containing the
+JIRA number of the issue you're working on as well as its title.
+
+For the problem described above, we might add the following:
+
+{% highlight scala %}
+// [SPARK-zz][CORE] Fix an issue
+ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SomeClass.this")
+{% endhighlight %}
+
+Otherwise, you will have to resolve those incompatibilies before opening or
+updating your pull request. Usually, the problems reported by MiMa are
+self-explanatory and revolve around missing members (methods or fields) that
+you will have to add back in order to maintain binary compatibility.
+
<h3>Checking Out Pull Requests</h3>
Git provides a mechanism for fetching remote pull requests into your own local repository.
@@ -181,7 +235,7 @@ It is due to an incorrect Scala library in the classpath. To fix it:
- Remove `scala-library-2.10.4.jar - lib_managed\jars`
In the event of "Could not find resource path for Web UI: org/apache/spark/ui/static",
-it's due to a classpath issue (some classes were probably not compiled). To fix this, it
+it's due to a classpath issue (some classes were probably not compiled). To fix this, it is
sufficient to run a test from the command line:
```
http://git-wip-us.apache.org/repos/asf/spark-website/blob/8b27c470/site/developer-tools.html
----------------------------------------------------------------------
diff --git a/site/developer-tools.html b/site/developer-tools.html
index d09815d..a44bfde 100644
--- a/site/developer-tools.html
+++ b/site/developer-tools.html
@@ -287,6 +287,55 @@ restart whenever <code>build/mvn</code> is called.</p>
<pre><code>build/mvn test -DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite test
</code></pre>
+<h3>Binary compatibility</h3>
+
+<p>To ensure binary compatibility, Spark uses <a href="https://github.com/typesafehub/migration-manager">MiMa</a>.</p>
+
+<h4>Ensuring binary compatibility</h4>
+
+<p>When working on an issue, it’s always a good idea to check that your changes do
+not introduce binary incompatibilities before opening a pull request.</p>
+
+<p>You can do so by running the following command:</p>
+
+<pre><code>$ dev/mima
+</code></pre>
+
+<p>A binary incompatibility reported by MiMa might look like the following:</p>
+
+<pre><code>[error] method this(org.apache.spark.sql.Dataset)Unit in class org.apache.spark.SomeClass does not have a correspondent in current version
+[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SomeClass.this")
+</code></pre>
+
+<p>If you open a pull request containing binary incompatibilities anyway, Jenkins
+will remind you by failing the test build with the following message:</p>
+
+<pre><code>Test build #xx has finished for PR yy at commit ffffff.
+
+ This patch fails MiMa tests.
+ This patch merges cleanly.
+ This patch adds no public classes.
+</code></pre>
+
+<h4>Solving a binary incompatibility</h4>
+
+<p>If you believe that your binary incompatibilies are justified or that MiMa
+reported false positives (e.g. the reported binary incompatibilities are about a
+non-user facing API), you can filter them out by adding an exclusion in
+<a href="https://github.com/apache/spark/blob/master/project/MimaExcludes.scala">project/MimaExcludes.scala</a>
+containing what was suggested by the MiMa report and a comment containing the
+JIRA number of the issue you’re working on as well as its title.</p>
+
+<p>For the problem described above, we might add the following:</p>
+
+<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="c1">// [SPARK-zz][CORE] Fix an issue</span>
+<span class="nc">ProblemFilters</span><span class="o">.</span><span class="n">exclude</span><span class="o">[</span><span class="kt">DirectMissingMethodProblem</span><span class="o">](</span><span class="s">"org.apache.spark.SomeClass.this"</span><span class="o">)</span></code></pre></figure>
+
+<p>Otherwise, you will have to resolve those incompatibilies before opening or
+updating your pull request. Usually, the problems reported by MiMa are
+self-explanatory and revolve around missing members (methods or fields) that
+you will have to add back in order to maintain binary compatibility.</p>
+
<h3>Checking Out Pull Requests</h3>
<p>Git provides a mechanism for fetching remote pull requests into your own local repository.
@@ -353,7 +402,7 @@ java.lang.NullPointerException
</ul>
<p>In the event of “Could not find resource path for Web UI: org/apache/spark/ui/static”,
-it’s due to a classpath issue (some classes were probably not compiled). To fix this, it
+it’s due to a classpath issue (some classes were probably not compiled). To fix this, it is
sufficient to run a test from the command line:</p>
<pre><code>build/sbt "test-only org.apache.spark.rdd.SortingSuite"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org