You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nick Pentreath (JIRA)" <ji...@apache.org> on 2016/05/11 11:56:12 UTC
[jira] [Comment Edited] (SPARK-13959) Audit MiMa excludes added in
SPARK-13948 to make sure none are unintended incompatibilities
[ https://issues.apache.org/jira/browse/SPARK-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280013#comment-15280013 ]
Nick Pentreath edited comment on SPARK-13959 at 5/11/16 11:55 AM:
------------------------------------------------------------------
All {{ML}} potential errors are related to {{DataFrame}} -> {{Dataset}} change:
{code}
[info] spark-mllib: found 4 potential binary incompatibilities while checking against org.apache.spark:spark-mllib_2.11:1.6.0 (filtered 151)
[error] * method transform(org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame in class org.apache.spark.ml.UnaryTransformer's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.sql.Dataset instead of (org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.UnaryTransformer.transform")
[error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel in class org.apache.spark.ml.classification.DecisionTreeClassifier's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.DecisionTreeClassifier.train")
[error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel in class org.apache.spark.ml.classification.LogisticRegression's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.LogisticRegression.train")
[error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel in class org.apache.spark.ml.regression.DecisionTreeRegressor's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.regression.DecisionTreeRegressor.train")
{code}
It's weird that the above errors for {{DecisionTreeClassifier.train}}, {{DecisionTreeRegressor.train}} and {{LogisticRegression.train}} refer to {{PredictionModel}} in the "current version". But in any case I verified the return types have not changed for those methods
was (Author: mlnick):
All {{ML}} potential errors are related to {{DataFrame}} -> {{Dataset}} change:
{code}
[info] spark-mllib: found 4 potential binary incompatibilities while checking against org.apache.spark:spark-mllib_2.11:1.6.0 (filtered 151)
[error] * method transform(org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame in class org.apache.spark.ml.UnaryTransformer's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.sql.Dataset instead of (org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.UnaryTransformer.transform")
[error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel in class org.apache.spark.ml.classification.DecisionTreeClassifier's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.DecisionTreeClassifier.train")
[error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel in class org.apache.spark.ml.classification.LogisticRegression's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.LogisticRegression.train")
[error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel in class org.apache.spark.ml.regression.DecisionTreeRegressor's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel
{code}
It's weird that the above errors for {{DecisionTreeClassifier.train}}, {{DecisionTreeRegressor.train}} and {{LogisticRegression.train}} refer to {{PredictionModel}} in the "current version". But in any case I verified the return types have not changed for those methods
> Audit MiMa excludes added in SPARK-13948 to make sure none are unintended incompatibilities
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-13959
> URL: https://issues.apache.org/jira/browse/SPARK-13959
> Project: Spark
> Issue Type: Task
> Components: Build
> Reporter: Josh Rosen
> Priority: Critical
>
> The patch for SPARK-13948 added a number of MiMa excludes for cases which were missed due to our old way of programatically generating excludes in GenerateMIMAIgnore.
> Before Spark 2.0, we need to audit the additional excludes that I added to make sure that none represent unintentional incompatibilities which should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org