[jira] [Issue Comment Deleted] (SPARK-20504) ML 2.2 QA: API: Java compatibility, docs

["Weichen Xu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-20504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Weichen Xu updated SPARK-20504:
-------------------------------
    Comment: was deleted

(was: You’re right this is really a headache. Java tools cannot extract several information `scalac` generated from the jars, such as the package private modifier, the private class modifier, and so on.
Through scala-doc to fetch and compare these change will be a terrible thing by programming processing I hope to avoid this way.
But these information was still reserved in the jar, I think we can get them through `scala reflection api`,
http://docs.scala-lang.org/overviews/reflection/overview.html
but I need to study them first.

Thanks!

Sent from Windows Mail

From: Joseph K. Bradley (JIRA)<ma...@apache.org>
Sent: ‎Tuesday‎, ‎May‎ ‎16‎, ‎2017 ‎7‎:‎24‎ ‎AM
To: WeichenXu123@outlook.com<ma...@outlook.com>


    [ https://issues.apache.org/jira/browse/SPARK-20504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011293#comment-16011293 ]

Joseph K. Bradley edited comment on SPARK-20504 at 5/15/17 11:23 PM:
---------------------------------------------------------------------

I found one issue: This won't pick up on methods which were package private in 2.1 and were made public in 2.2.  E.g.: Matrix.foreachActive was made public in [SPARK-17471] here: https://github.com/apache/spark/pull/15628/files#diff-440e1b707197e577b932a055ab16293eR158
But it does not show up in the diff.  (Update: This foreachActive API will be OK to leave as is.)

We won't be able to identify these cases from the JARs; we'll have to rely on the docs.



was (Author: josephkb):
I found one issue: This won't pick up on methods which were package private in 2.1 and were made public in 2.2.  E.g.: Matrix.foreachActive was made public in [SPARK-17471] here: https://github.com/apache/spark/pull/15628/files#diff-440e1b707197e577b932a055ab16293eR158
But it does not show up in the diff.  (I'll ping on that PR to see what people want to do.)

We won't be able to identify these cases from the JARs; we'll have to rely on the docs.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
)

> ML 2.2 QA: API: Java compatibility, docs
> ----------------------------------------
>
>                 Key: SPARK-20504
>                 URL: https://issues.apache.org/jira/browse/SPARK-20504
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Documentation, Java API, ML, MLlib
>            Reporter: Joseph K. Bradley
>            Assignee: Weichen Xu
>            Priority: Blocker
>             Fix For: 2.2.0
>
>         Attachments: 1_process_script.sh, 2_signature.diff, 3_added_class_signature, 4_common_ml_class, 5_added_ml_class
>
>
> Check Java compatibility for this release:
> * APIs in {{spark.ml}}
> * New APIs in {{spark.mllib}} (There should be few, if any.)
> Checking compatibility means:
> * Checking for differences in how Scala and Java handle types. Some items to look out for are:
> ** Check for generic "Object" types where Java cannot understand complex Scala types.
> *** *Note*: The Java docs do not always match the bytecode. If you find a problem, please verify it using {{javap}}.
> ** Check Scala objects (especially with nesting!) carefully.  These may not be understood in Java, or they may be accessible only via the weirdly named Java types (with "$" or "#") which are generated by the Scala compiler.
> ** Check for uses of Scala and Java enumerations, which can show up oddly in the other language's doc.  (In {{spark.ml}}, we have largely tried to avoid using enumerations, and have instead favored plain strings.)
> * Check for differences in generated Scala vs Java docs.  E.g., one past issue was that Javadocs did not respect Scala's package private modifier.
> If you find issues, please comment here, or for larger items, create separate JIRAs and link here as "requires".
> * Remember that we should not break APIs from previous releases.  If you find a problem, check if it was introduced in this Spark release (in which case we can fix it) or in a previous one (in which case we can create a java-friendly version of the API).
> * If needed for complex issues, create small Java unit tests which execute each method.  (Algorithmic correctness can be checked in Scala.)
> Recommendations for how to complete this task:
> * There are not great tools.  In the past, this task has been done by:
> ** Generating API docs
> ** Building JAR and outputting the Java class signatures for MLlib
> ** Manually inspecting and searching the docs and class signatures for issues
> * If you do have ideas for better tooling, please say so we can make this task easier in the future!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org