[jira] [Resolved] (SPARK-2709) Add a tool for certifying Spark API compatiblity

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-2709.
------------------------------
    Resolution: Won't Fix

It sounds like this won't happen / didn't happen, but reopen if there's an intent to open source it.

> Add a tool for certifying Spark API compatiblity
> ------------------------------------------------
>
>                 Key: SPARK-2709
>                 URL: https://issues.apache.org/jira/browse/SPARK-2709
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>            Reporter: Patrick Wendell
>            Assignee: Prashant Sharma
>            Priority: Critical
>
> As Spark is packaged by more and more distributors, it would be good to have a tool that verifies API compatiblity of a provided Spark package. The tool would certify that a vendor distrubtion of Spark contains all of the API's present in a particular upstream Spark version.
> This will help vendors make sure they remain "API compliant" when they make changes or back ports to Spark. It will also discourage vendors from knowingly breaking API's, because anyone can audit their distribution and see that they have removed support for certain API's.
> I'm hoping a tool like this will avoid API fragmentation in the Spark community.
> One "poor man's" implementation of this is that a vendor can just run the binary compatibility checks in the spark build against an upstream version of Spark. That's a pretty good start, but it means you can't come as a third party and audit a distribution.
> Another approach would be to have something where anyone can come in and audit a distribution even if they don't have access to the packaging and source code. That would look something like this:
> 1. For each release we publish a manifest of all public API's (we might borrow the MIMA string representation of bye code signatures)
> 2. We package an auditing tool as a jar file.
> 3. The user runs a tool with spark-submit that reflectively walks through all exposed Spark API's and makes sure that everything on the manifest is encountered.
> From the implementation side, this is just brainstorming at this point.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org