You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2023/03/14 23:31:50 UTC
[spark] branch master updated: [SPARK-42508][CONNECT][ML] Extract the common .ml classes to `mllib-common`
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5a5f04c4880 [SPARK-42508][CONNECT][ML] Extract the common .ml classes to `mllib-common`
5a5f04c4880 is described below
commit 5a5f04c488074ec159be77079647c971fc32eb68
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Mar 15 08:31:32 2023 +0900
[SPARK-42508][CONNECT][ML] Extract the common .ml classes to `mllib-common`
### What changes were proposed in this pull request?
## Common classes
This PR move following packages/classes into new module `spark-mllib-common`:
1, `org.apache.spark.ml.attribute.*`
2, `org.apache.spark.ml.linalg.*`
3, `org.apache.spark.ml.param.*` except:
- `SharedParamsCodeGen` due to it depends on `spark-core`, and it is not a runtime dependency;
- `getExecutionContext` method in `HasParallelism` due to it depends on `spark-core`;
4, `org.apache.spark.ml.PredictorParams`
5, `org.apache.spark.ml.classification.ClassifierParams`
6, `org.apache.spark.ml.classification.ProbabilisticClassifierParams`
7, `org.apache.spark.ml.util.Identifiable`
8, `org.apache.spark.ml.util.SchemaUtils`
9, `org.apache.spark.ml.feature.{LabeledPoint, Instance}`
When implementing the algorithms, we should also move corresponding Params traits into `spark-mllib-common`, e.g. `DecisionTreeParams`, `DecisionTreeClassifierParams`, etc
## Testsuites
This PR moves following testsuites to `spark-mllib-common`:
1, `org.apache.spark.ml.attribute.*`
2, `org.apache.spark.ml.linalg.*` except `test("JavaTypeInference with VectorUDT")` in `VectorUDTSuite` due to cyclical dependency;
3, `org.apache.spark.ml.param.*` except `test("Filtering ParamMap")` in `ParamsSuite` due to cyclical dependency;
4, `org.apache.spark.ml.util.IdentifiableSuite`
## Dependency
The new module `spark-mllib-common` depends on following packages:
1, `json4s-jackson`
2, `spark-tags`
3, `spark-mllib-local`
4, `spark-catalyst`:
- `org.apache.spark.sql.types.*`
- `org.apache.spark.sql.catalyst.InternalRow` in `VectorUDT` and `MatrixUDT`
- `org.apache.spark.sql.catalyst.expressions.{GenericInternalRow, UnsafeArrayData}` in `VectorUDT` and `MatrixUDT`
## Dependency changes
This PR make `spark-mllib` depend on the `spark-mllib-common`;
This PR make Spark Connect Scala Client `spark-connect-client-jvm` depend on the `spark-mllib-common`;
This PR make Spark Connect Server `spark-connect` depend on the `spark-mllib`;
## Reorg mllib modules
This PR also reorg the mllib modules:
```
mllib
| --- core: spark-mllib
| --- local: spark-mllib-local
| --- common: spark-mllib-common
```
### Why are the changes needed?
to support ML APIs atop Spark-Connect Scala Client
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
updated tests
Closes #40097 from zhengruifeng/connect_decision_tree.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.github/workflows/build_and_test.yml | 2 +-
connector/connect/client/jvm/pom.xml | 12 ++
.../main/scala/org/apache/spark/ml/Estimator.scala | 55 ++++---
.../src/main/scala/org/apache/spark/ml/Model.scala | 11 +-
.../main/scala/org/apache/spark/ml/Pipeline.scala | 63 ++++++++
.../main/scala/org/apache/spark/ml/Predictor.scala | 164 +++++++++++++++++++++
.../scala/org/apache/spark/ml/Transformer.scala | 54 ++++---
.../spark/ml/classification/Classifier.scala | 138 +++++++++++++++++
.../classification/ProbabilisticClassifier.scala | 126 ++++++++++++++++
connector/connect/server/pom.xml | 12 ++
dev/sparktestsupport/modules.py | 41 ++++--
dev/sparktestsupport/utils.py | 19 ++-
{mllib-local => mllib/common}/pom.xml | 52 ++++---
.../main/scala/org/apache/spark/ml/Predictor.scala | 66 +++++++++
.../apache/spark/ml/attribute/AttributeGroup.scala | 0
.../apache/spark/ml/attribute/AttributeKeys.scala | 0
.../apache/spark/ml/attribute/AttributeType.scala | 0
.../org/apache/spark/ml/attribute/attributes.scala | 3 +-
.../apache/spark/ml/attribute/package-info.java | 0
.../org/apache/spark/ml/attribute/package.scala | 0
.../spark/ml/classification/Classifier.scala} | 40 ++---
.../classification/ProbabilisticClassifier.scala} | 41 ++----
.../org/apache/spark/ml/feature/Instance.scala} | 12 +-
.../org/apache/spark/ml/feature/LabeledPoint.scala | 0
.../spark/ml/linalg/JsonMatrixConverter.scala | 0
.../spark/ml/linalg/JsonVectorConverter.scala | 0
.../org/apache/spark/ml/linalg/MatrixUDT.scala | 2 +-
.../org/apache/spark/ml/linalg/SQLDataTypes.scala | 0
.../org/apache/spark/ml/linalg/VectorUDT.scala | 2 +-
.../scala/org/apache/spark/ml/param/params.scala | 0
.../spark/ml/param/shared/HasParallelism.scala | 19 +--
.../spark/ml/param/shared/sharedParams.scala | 0
.../org/apache/spark/ml/util/Identifiable.scala | 0
.../org/apache/spark/ml/util/SchemaUtils.scala | 0
.../spark/ml/attribute/AttributeGroupSuite.scala | 0
.../apache/spark/ml/attribute/AttributeSuite.scala | 0
.../spark/ml/linalg/JsonMatrixConverterSuite.scala | 0
.../spark/ml/linalg/JsonVectorConverterSuite.scala | 0
.../apache/spark/ml/linalg/MatrixUDTSuite.scala | 0
.../apache/spark/ml/linalg/SQLDataTypesSuite.scala | 0
.../apache/spark/ml/linalg/VectorUDTSuite.scala | 9 --
.../org/apache/spark/ml/param/ParamsSuite.scala | 78 ----------
.../org/apache/spark/ml/param/TestParams.scala | 0
.../spark/ml/param/shared/SharedParamsSuite.scala | 0
.../apache/spark/ml/util/IdentifiableSuite.scala | 0
.../UDTSerializationBenchmark-jdk11-results.txt | 0
.../UDTSerializationBenchmark-jdk17-results.txt | 0
.../UDTSerializationBenchmark-results.txt | 0
mllib/{ => core}/pom.xml | 14 +-
.../org.apache.spark.ml.util.MLFormatRegister | 0
...org.apache.spark.sql.sources.DataSourceRegister | 0
.../org/apache/spark/ml/feature/stopwords/README | 0
.../apache/spark/ml/feature/stopwords/danish.txt | 0
.../apache/spark/ml/feature/stopwords/dutch.txt | 0
.../apache/spark/ml/feature/stopwords/english.txt | 0
.../apache/spark/ml/feature/stopwords/finnish.txt | 0
.../apache/spark/ml/feature/stopwords/french.txt | 0
.../apache/spark/ml/feature/stopwords/german.txt | 0
.../spark/ml/feature/stopwords/hungarian.txt | 0
.../apache/spark/ml/feature/stopwords/italian.txt | 0
.../spark/ml/feature/stopwords/norwegian.txt | 0
.../spark/ml/feature/stopwords/portuguese.txt | 0
.../apache/spark/ml/feature/stopwords/russian.txt | 0
.../apache/spark/ml/feature/stopwords/spanish.txt | 0
.../apache/spark/ml/feature/stopwords/swedish.txt | 0
.../apache/spark/ml/feature/stopwords/turkish.txt | 0
.../main/scala/org/apache/spark/ml/Estimator.scala | 0
.../src/main/scala/org/apache/spark/ml/Model.scala | 0
.../main/scala/org/apache/spark/ml/Pipeline.scala | 0
.../main/scala/org/apache/spark/ml/Predictor.scala | 36 -----
.../scala/org/apache/spark/ml/Transformer.scala | 0
.../scala/org/apache/spark/ml/ann/BreezeUtil.scala | 0
.../main/scala/org/apache/spark/ml/ann/Layer.scala | 0
.../org/apache/spark/ml/ann/LossFunction.scala | 0
.../ml/classification/ClassificationSummary.scala | 0
.../spark/ml/classification/Classifier.scala | 22 +--
.../ml/classification/DecisionTreeClassifier.scala | 0
.../spark/ml/classification/FMClassifier.scala | 0
.../spark/ml/classification/GBTClassifier.scala | 0
.../apache/spark/ml/classification/LinearSVC.scala | 0
.../ml/classification/LogisticRegression.scala | 0
.../MultilayerPerceptronClassifier.scala | 0
.../spark/ml/classification/NaiveBayes.scala | 0
.../apache/spark/ml/classification/OneVsRest.scala | 5 +-
.../classification/ProbabilisticClassifier.scala | 20 +--
.../ml/classification/RandomForestClassifier.scala | 0
.../spark/ml/clustering/BisectingKMeans.scala | 0
.../spark/ml/clustering/ClusteringSummary.scala | 0
.../spark/ml/clustering/GaussianMixture.scala | 0
.../org/apache/spark/ml/clustering/KMeans.scala | 0
.../scala/org/apache/spark/ml/clustering/LDA.scala | 0
.../ml/clustering/PowerIterationClustering.scala | 0
.../evaluation/BinaryClassificationEvaluator.scala | 0
.../spark/ml/evaluation/ClusteringEvaluator.scala | 0
.../spark/ml/evaluation/ClusteringMetrics.scala | 0
.../org/apache/spark/ml/evaluation/Evaluator.scala | 0
.../MulticlassClassificationEvaluator.scala | 0
.../MultilabelClassificationEvaluator.scala | 0
.../spark/ml/evaluation/RankingEvaluator.scala | 0
.../spark/ml/evaluation/RegressionEvaluator.scala | 0
.../main/scala/org/apache/spark/ml/events.scala | 0
.../org/apache/spark/ml/feature/Binarizer.scala | 0
.../ml/feature/BucketedRandomProjectionLSH.scala | 0
.../org/apache/spark/ml/feature/Bucketizer.scala | 0
.../apache/spark/ml/feature/ChiSqSelector.scala | 0
.../apache/spark/ml/feature/CountVectorizer.scala | 0
.../scala/org/apache/spark/ml/feature/DCT.scala | 0
.../spark/ml/feature/ElementwiseProduct.scala | 0
.../apache/spark/ml/feature/FeatureHasher.scala | 0
.../org/apache/spark/ml/feature/HashingTF.scala | 0
.../scala/org/apache/spark/ml/feature/IDF.scala | 0
.../org/apache/spark/ml/feature/Imputer.scala | 0
.../org/apache/spark/ml/feature/Instance.scala | 10 --
.../org/apache/spark/ml/feature/Interaction.scala | 0
.../scala/org/apache/spark/ml/feature/LSH.scala | 0
.../org/apache/spark/ml/feature/MaxAbsScaler.scala | 0
.../org/apache/spark/ml/feature/MinHashLSH.scala | 0
.../org/apache/spark/ml/feature/MinMaxScaler.scala | 0
.../scala/org/apache/spark/ml/feature/NGram.scala | 0
.../org/apache/spark/ml/feature/Normalizer.scala | 0
.../apache/spark/ml/feature/OneHotEncoder.scala | 0
.../scala/org/apache/spark/ml/feature/PCA.scala | 0
.../spark/ml/feature/PolynomialExpansion.scala | 0
.../spark/ml/feature/QuantileDiscretizer.scala | 0
.../org/apache/spark/ml/feature/RFormula.scala | 0
.../apache/spark/ml/feature/RFormulaParser.scala | 0
.../org/apache/spark/ml/feature/RobustScaler.scala | 0
.../apache/spark/ml/feature/SQLTransformer.scala | 0
.../org/apache/spark/ml/feature/Selector.scala | 0
.../apache/spark/ml/feature/StandardScaler.scala | 0
.../apache/spark/ml/feature/StopWordsRemover.scala | 0
.../apache/spark/ml/feature/StringIndexer.scala | 0
.../org/apache/spark/ml/feature/Tokenizer.scala | 0
.../ml/feature/UnivariateFeatureSelector.scala | 0
.../ml/feature/VarianceThresholdSelector.scala | 0
.../apache/spark/ml/feature/VectorAssembler.scala | 0
.../apache/spark/ml/feature/VectorIndexer.scala | 0
.../apache/spark/ml/feature/VectorSizeHint.scala | 0
.../org/apache/spark/ml/feature/VectorSlicer.scala | 0
.../org/apache/spark/ml/feature/Word2Vec.scala | 0
.../org/apache/spark/ml/feature/package-info.java | 0
.../org/apache/spark/ml/feature/package.scala | 0
.../scala/org/apache/spark/ml/fpm/FPGrowth.scala | 0
.../scala/org/apache/spark/ml/fpm/PrefixSpan.scala | 0
.../main/scala/org/apache/spark/ml/functions.scala | 0
.../org/apache/spark/ml/image/HadoopUtils.scala | 0
.../org/apache/spark/ml/image/ImageSchema.scala | 0
.../optim/IterativelyReweightedLeastSquares.scala | 0
.../spark/ml/optim/NormalEquationSolver.scala | 0
.../spark/ml/optim/WeightedLeastSquares.scala | 0
.../ml/optim/aggregator/AFTBlockAggregator.scala | 0
.../aggregator/BinaryLogisticBlockAggregator.scala | 0
.../aggregator/DifferentiableLossAggregator.scala | 0
.../ml/optim/aggregator/HingeBlockAggregator.scala | 0
.../ml/optim/aggregator/HuberBlockAggregator.scala | 0
.../aggregator/LeastSquaresBlockAggregator.scala | 0
.../MultinomialLogisticBlockAggregator.scala | 0
.../optim/loss/DifferentiableRegularization.scala | 0
.../spark/ml/optim/loss/RDDLossFunction.scala | 0
.../scala/org/apache/spark/ml/package-info.java | 0
.../main/scala/org/apache/spark/ml/package.scala | 0
.../ml/param/shared/HasExecutionContext.scala} | 21 +--
.../ml/param/shared/SharedParamsCodeGen.scala | 0
.../scala/org/apache/spark/ml/python/MLSerDe.scala | 0
.../spark/ml/r/AFTSurvivalRegressionWrapper.scala | 0
.../scala/org/apache/spark/ml/r/ALSWrapper.scala | 0
.../apache/spark/ml/r/BisectingKMeansWrapper.scala | 0
.../spark/ml/r/DecisionTreeClassifierWrapper.scala | 0
.../spark/ml/r/DecisionTreeRegressorWrapper.scala | 0
.../apache/spark/ml/r/FMClassifierWrapper.scala | 0
.../org/apache/spark/ml/r/FMRegressorWrapper.scala | 0
.../org/apache/spark/ml/r/FPGrowthWrapper.scala | 0
.../apache/spark/ml/r/GBTClassifierWrapper.scala | 0
.../apache/spark/ml/r/GBTRegressorWrapper.scala | 0
.../apache/spark/ml/r/GaussianMixtureWrapper.scala | 0
.../ml/r/GeneralizedLinearRegressionWrapper.scala | 0
.../spark/ml/r/IsotonicRegressionWrapper.scala | 0
.../org/apache/spark/ml/r/KMeansWrapper.scala | 0
.../org/apache/spark/ml/r/KSTestWrapper.scala | 0
.../scala/org/apache/spark/ml/r/LDAWrapper.scala | 0
.../spark/ml/r/LinearRegressionWrapper.scala | 0
.../org/apache/spark/ml/r/LinearSVCWrapper.scala | 0
.../spark/ml/r/LogisticRegressionWrapper.scala | 0
.../r/MultilayerPerceptronClassifierWrapper.scala | 0
.../org/apache/spark/ml/r/NaiveBayesWrapper.scala | 0
.../ml/r/PowerIterationClusteringWrapper.scala | 0
.../org/apache/spark/ml/r/PrefixSpanWrapper.scala | 0
.../org/apache/spark/ml/r/RWrapperUtils.scala | 0
.../scala/org/apache/spark/ml/r/RWrappers.scala | 0
.../spark/ml/r/RandomForestClassifierWrapper.scala | 0
.../spark/ml/r/RandomForestRegressorWrapper.scala | 0
.../org/apache/spark/ml/recommendation/ALS.scala | 0
.../ml/regression/AFTSurvivalRegression.scala | 0
.../ml/regression/DecisionTreeRegressor.scala | 0
.../apache/spark/ml/regression/FMRegressor.scala | 0
.../apache/spark/ml/regression/GBTRegressor.scala | 0
.../regression/GeneralizedLinearRegression.scala | 0
.../spark/ml/regression/IsotonicRegression.scala | 0
.../spark/ml/regression/LinearRegression.scala | 0
.../ml/regression/RandomForestRegressor.scala | 0
.../org/apache/spark/ml/regression/Regressor.scala | 0
.../spark/ml/source/image/ImageDataSource.scala | 0
.../spark/ml/source/image/ImageFileFormat.scala | 0
.../spark/ml/source/image/ImageOptions.scala | 0
.../spark/ml/source/libsvm/LibSVMDataSource.scala | 0
.../spark/ml/source/libsvm/LibSVMOptions.scala | 0
.../spark/ml/source/libsvm/LibSVMRelation.scala | 0
.../scala/org/apache/spark/ml/stat/ANOVATest.scala | 0
.../org/apache/spark/ml/stat/ChiSquareTest.scala | 0
.../org/apache/spark/ml/stat/Correlation.scala | 0
.../org/apache/spark/ml/stat/FValueTest.scala | 0
.../spark/ml/stat/KolmogorovSmirnovTest.scala | 0
.../spark/ml/stat/MultiClassSummarizer.scala | 0
.../org/apache/spark/ml/stat/Summarizer.scala | 0
.../main/scala/org/apache/spark/ml/tree/Node.scala | 0
.../scala/org/apache/spark/ml/tree/Split.scala | 0
.../apache/spark/ml/tree/impl/BaggedPoint.scala | 0
.../spark/ml/tree/impl/DTStatsAggregator.scala | 0
.../spark/ml/tree/impl/DecisionTreeMetadata.scala | 0
.../spark/ml/tree/impl/GradientBoostedTrees.scala | 0
.../apache/spark/ml/tree/impl/RandomForest.scala | 0
.../apache/spark/ml/tree/impl/TimeTracker.scala | 0
.../org/apache/spark/ml/tree/impl/TreePoint.scala | 0
.../org/apache/spark/ml/tree/treeModels.scala | 0
.../org/apache/spark/ml/tree/treeParams.scala | 0
.../apache/spark/ml/tuning/CrossValidator.scala | 4 +-
.../apache/spark/ml/tuning/ParamGridBuilder.scala | 0
.../spark/ml/tuning/TrainValidationSplit.scala | 4 +-
.../apache/spark/ml/tuning/ValidatorParams.scala | 0
.../org/apache/spark/ml/util/DatasetUtils.scala | 0
.../apache/spark/ml/util/HasTrainingSummary.scala | 0
.../org/apache/spark/ml/util/Instrumentation.scala | 0
.../org/apache/spark/ml/util/MetadataUtils.scala | 0
.../scala/org/apache/spark/ml/util/ReadWrite.scala | 0
.../org/apache/spark/ml/util/stopwatches.scala | 0
.../scala/org/apache/spark/mllib/JavaPackage.java | 0
.../mllib/api/python/FPGrowthModelWrapper.scala | 0
.../api/python/GaussianMixtureModelWrapper.scala | 0
.../spark/mllib/api/python/LDAModelWrapper.scala | 0
.../python/MatrixFactorizationModelWrapper.scala | 0
.../PowerIterationClusteringModelWrapper.scala | 0
.../mllib/api/python/PrefixSpanModelWrapper.scala | 0
.../spark/mllib/api/python/PythonMLLibAPI.scala | 0
.../mllib/api/python/Word2VecModelWrapper.scala | 0
.../apache/spark/mllib/api/python/package.scala | 0
.../mllib/classification/ClassificationModel.scala | 0
.../mllib/classification/LogisticRegression.scala | 0
.../spark/mllib/classification/NaiveBayes.scala | 0
.../apache/spark/mllib/classification/SVM.scala | 0
.../StreamingLogisticRegressionWithSGD.scala | 0
.../impl/GLMClassificationModel.scala | 0
.../spark/mllib/clustering/BisectingKMeans.scala | 0
.../mllib/clustering/BisectingKMeansModel.scala | 0
.../spark/mllib/clustering/DistanceMeasure.scala | 0
.../spark/mllib/clustering/GaussianMixture.scala | 0
.../mllib/clustering/GaussianMixtureModel.scala | 0
.../org/apache/spark/mllib/clustering/KMeans.scala | 0
.../spark/mllib/clustering/KMeansModel.scala | 0
.../org/apache/spark/mllib/clustering/LDA.scala | 0
.../apache/spark/mllib/clustering/LDAModel.scala | 0
.../spark/mllib/clustering/LDAOptimizer.scala | 0
.../apache/spark/mllib/clustering/LDAUtils.scala | 0
.../spark/mllib/clustering/LocalKMeans.scala | 0
.../clustering/PowerIterationClustering.scala | 0
.../spark/mllib/clustering/StreamingKMeans.scala | 0
.../spark/mllib/evaluation/AreaUnderCurve.scala | 0
.../evaluation/BinaryClassificationMetrics.scala | 0
.../spark/mllib/evaluation/MulticlassMetrics.scala | 0
.../spark/mllib/evaluation/MultilabelMetrics.scala | 0
.../spark/mllib/evaluation/RankingMetrics.scala | 0
.../spark/mllib/evaluation/RegressionMetrics.scala | 0
.../BinaryClassificationMetricComputers.scala | 0
.../evaluation/binary/BinaryConfusionMatrix.scala | 0
.../evaluation/binary/BinaryLabelCounter.scala | 0
.../apache/spark/mllib/feature/ChiSqSelector.scala | 0
.../spark/mllib/feature/ElementwiseProduct.scala | 0
.../org/apache/spark/mllib/feature/HashingTF.scala | 0
.../scala/org/apache/spark/mllib/feature/IDF.scala | 0
.../apache/spark/mllib/feature/Normalizer.scala | 0
.../scala/org/apache/spark/mllib/feature/PCA.scala | 0
.../spark/mllib/feature/StandardScaler.scala | 0
.../spark/mllib/feature/VectorTransformer.scala | 0
.../org/apache/spark/mllib/feature/Word2Vec.scala | 0
.../apache/spark/mllib/fpm/AssociationRules.scala | 0
.../org/apache/spark/mllib/fpm/FPGrowth.scala | 0
.../scala/org/apache/spark/mllib/fpm/FPTree.scala | 0
.../apache/spark/mllib/fpm/LocalPrefixSpan.scala | 0
.../org/apache/spark/mllib/fpm/PrefixSpan.scala | 0
.../org/apache/spark/mllib/linalg/ARPACK.scala | 0
.../scala/org/apache/spark/mllib/linalg/BLAS.scala | 0
.../spark/mllib/linalg/CholeskyDecomposition.scala | 0
.../mllib/linalg/EigenValueDecomposition.scala | 0
.../org/apache/spark/mllib/linalg/LAPACK.scala | 0
.../org/apache/spark/mllib/linalg/Matrices.scala | 0
.../mllib/linalg/SingularValueDecomposition.scala | 0
.../org/apache/spark/mllib/linalg/Vectors.scala | 0
.../mllib/linalg/distributed/BlockMatrix.scala | 0
.../linalg/distributed/CoordinateMatrix.scala | 0
.../linalg/distributed/DistributedMatrix.scala | 0
.../linalg/distributed/IndexedRowMatrix.scala | 0
.../spark/mllib/linalg/distributed/RowMatrix.scala | 0
.../apache/spark/mllib/optimization/Gradient.scala | 0
.../spark/mllib/optimization/GradientDescent.scala | 0
.../apache/spark/mllib/optimization/LBFGS.scala | 0
.../org/apache/spark/mllib/optimization/NNLS.scala | 0
.../spark/mllib/optimization/Optimizer.scala | 0
.../apache/spark/mllib/optimization/Updater.scala | 0
.../scala/org/apache/spark/mllib/package-info.java | 0
.../scala/org/apache/spark/mllib/package.scala | 0
.../apache/spark/mllib/pmml/PMMLExportable.scala | 0
.../BinaryClassificationPMMLModelExport.scala | 0
.../export/GeneralizedLinearPMMLModelExport.scala | 0
.../mllib/pmml/export/KMeansPMMLModelExport.scala | 0
.../spark/mllib/pmml/export/PMMLModelExport.scala | 0
.../mllib/pmml/export/PMMLModelExportFactory.scala | 0
.../spark/mllib/random/RandomDataGenerator.scala | 0
.../org/apache/spark/mllib/random/RandomRDDs.scala | 0
.../spark/mllib/rdd/MLPairRDDFunctions.scala | 0
.../org/apache/spark/mllib/rdd/RDDFunctions.scala | 0
.../org/apache/spark/mllib/rdd/RandomRDD.scala | 0
.../org/apache/spark/mllib/rdd/SlidingRDD.scala | 0
.../apache/spark/mllib/recommendation/ALS.scala | 0
.../recommendation/MatrixFactorizationModel.scala | 0
.../regression/GeneralizedLinearAlgorithm.scala | 0
.../mllib/regression/IsotonicRegression.scala | 0
.../spark/mllib/regression/LabeledPoint.scala | 0
.../org/apache/spark/mllib/regression/Lasso.scala | 0
.../spark/mllib/regression/LinearRegression.scala | 0
.../spark/mllib/regression/RegressionModel.scala | 0
.../spark/mllib/regression/RidgeRegression.scala | 0
.../regression/StreamingLinearAlgorithm.scala | 0
.../StreamingLinearRegressionWithSGD.scala | 0
.../mllib/regression/impl/GLMRegressionModel.scala | 0
.../apache/spark/mllib/stat/KernelDensity.scala | 0
.../mllib/stat/MultivariateOnlineSummarizer.scala | 0
.../stat/MultivariateStatisticalSummary.scala | 0
.../org/apache/spark/mllib/stat/Statistics.scala | 0
.../spark/mllib/stat/correlation/Correlation.scala | 0
.../stat/correlation/PearsonCorrelation.scala | 0
.../stat/correlation/SpearmanCorrelation.scala | 0
.../stat/distribution/MultivariateGaussian.scala | 0
.../apache/spark/mllib/stat/test/ChiSqTest.scala | 0
.../mllib/stat/test/KolmogorovSmirnovTest.scala | 0
.../spark/mllib/stat/test/StreamingTest.scala | 0
.../mllib/stat/test/StreamingTestMethod.scala | 0
.../apache/spark/mllib/stat/test/TestResult.scala | 0
.../org/apache/spark/mllib/tree/DecisionTree.scala | 0
.../spark/mllib/tree/GradientBoostedTrees.scala | 0
.../org/apache/spark/mllib/tree/RandomForest.scala | 0
.../spark/mllib/tree/configuration/Algo.scala | 0
.../tree/configuration/BoostingStrategy.scala | 0
.../configuration/EnsembleCombiningStrategy.scala | 0
.../mllib/tree/configuration/FeatureType.scala | 0
.../tree/configuration/QuantileStrategy.scala | 0
.../spark/mllib/tree/configuration/Strategy.scala | 0
.../apache/spark/mllib/tree/impurity/Entropy.scala | 0
.../apache/spark/mllib/tree/impurity/Gini.scala | 0
.../spark/mllib/tree/impurity/Impurities.scala | 0
.../spark/mllib/tree/impurity/Impurity.scala | 0
.../spark/mllib/tree/impurity/Variance.scala | 0
.../spark/mllib/tree/loss/AbsoluteError.scala | 0
.../org/apache/spark/mllib/tree/loss/LogLoss.scala | 0
.../org/apache/spark/mllib/tree/loss/Loss.scala | 0
.../org/apache/spark/mllib/tree/loss/Losses.scala | 0
.../spark/mllib/tree/loss/SquaredError.scala | 0
.../spark/mllib/tree/model/DecisionTreeModel.scala | 0
.../mllib/tree/model/InformationGainStats.scala | 0
.../org/apache/spark/mllib/tree/model/Node.scala | 0
.../apache/spark/mllib/tree/model/Predict.scala | 0
.../org/apache/spark/mllib/tree/model/Split.scala | 0
.../mllib/tree/model/treeEnsembleModels.scala | 0
.../org/apache/spark/mllib/tree/package.scala | 0
.../apache/spark/mllib/util/DataValidators.scala | 0
.../spark/mllib/util/KMeansDataGenerator.scala | 0
.../spark/mllib/util/LinearDataGenerator.scala | 0
.../util/LogisticRegressionDataGenerator.scala | 0
.../apache/spark/mllib/util/MFDataGenerator.scala | 0
.../org/apache/spark/mllib/util/MLUtils.scala | 0
.../apache/spark/mllib/util/NumericParser.scala | 0
.../apache/spark/mllib/util/SVMDataGenerator.scala | 0
.../apache/spark/mllib/util/modelSaveLoad.scala | 0
.../java/org/apache/spark/SharedSparkSession.java | 0
.../org/apache/spark/ml/JavaPipelineSuite.java | 0
.../ml/attribute/JavaAttributeGroupSuite.java | 0
.../spark/ml/attribute/JavaAttributeSuite.java | 0
.../JavaDecisionTreeClassifierSuite.java | 0
.../ml/classification/JavaGBTClassifierSuite.java | 0
.../JavaLogisticRegressionSuite.java | 0
.../JavaMultilayerPerceptronClassifierSuite.java | 0
.../ml/classification/JavaNaiveBayesSuite.java | 0
.../ml/classification/JavaOneVsRestSuite.java | 0
.../JavaRandomForestClassifierSuite.java | 0
.../spark/ml/clustering/JavaKMeansSuite.java | 0
.../spark/ml/feature/JavaBucketizerSuite.java | 0
.../org/apache/spark/ml/feature/JavaDCTSuite.java | 0
.../spark/ml/feature/JavaHashingTFSuite.java | 0
.../spark/ml/feature/JavaNormalizerSuite.java | 0
.../org/apache/spark/ml/feature/JavaPCASuite.java | 0
.../ml/feature/JavaPolynomialExpansionSuite.java | 0
.../spark/ml/feature/JavaStandardScalerSuite.java | 0
.../ml/feature/JavaStopWordsRemoverSuite.java | 0
.../spark/ml/feature/JavaStringIndexerSuite.java | 0
.../spark/ml/feature/JavaTokenizerSuite.java | 0
.../spark/ml/feature/JavaVectorAssemblerSuite.java | 0
.../spark/ml/feature/JavaVectorIndexerSuite.java | 0
.../spark/ml/feature/JavaVectorSlicerSuite.java | 0
.../apache/spark/ml/feature/JavaWord2VecSuite.java | 0
.../spark/ml/linalg/JavaSQLDataTypesSuite.java | 0
.../org/apache/spark/ml/param/JavaParamsSuite.java | 0
.../org/apache/spark/ml/param/JavaTestParams.java | 0
.../regression/JavaDecisionTreeRegressorSuite.java | 0
.../spark/ml/regression/JavaGBTRegressorSuite.java | 0
.../ml/regression/JavaLinearRegressionSuite.java | 0
.../regression/JavaRandomForestRegressorSuite.java | 0
.../ml/source/libsvm/JavaLibSVMRelationSuite.java | 0
.../ml/stat/JavaKolmogorovSmirnovTestSuite.java | 0
.../apache/spark/ml/stat/JavaSummarizerSuite.java | 0
.../spark/ml/tuning/JavaCrossValidatorSuite.java | 0
.../spark/ml/util/JavaDefaultReadWriteSuite.java | 0
.../JavaLogisticRegressionSuite.java | 0
.../mllib/classification/JavaNaiveBayesSuite.java | 0
.../spark/mllib/classification/JavaSVMSuite.java | 0
.../JavaStreamingLogisticRegressionSuite.java | 0
.../mllib/clustering/JavaBisectingKMeansSuite.java | 0
.../mllib/clustering/JavaGaussianMixtureSuite.java | 0
.../spark/mllib/clustering/JavaKMeansSuite.java | 0
.../spark/mllib/clustering/JavaLDASuite.java | 0
.../mllib/clustering/JavaStreamingKMeansSuite.java | 0
.../mllib/evaluation/JavaRankingMetricsSuite.java | 0
.../apache/spark/mllib/feature/JavaTfIdfSuite.java | 0
.../spark/mllib/feature/JavaWord2VecSuite.java | 0
.../spark/mllib/fpm/JavaAssociationRulesSuite.java | 0
.../apache/spark/mllib/fpm/JavaFPGrowthSuite.java | 0
.../spark/mllib/fpm/JavaPrefixSpanSuite.java | 0
.../spark/mllib/linalg/JavaMatricesSuite.java | 0
.../spark/mllib/linalg/JavaVectorsSuite.java | 0
.../linalg/distributed/JavaRowMatrixSuite.java | 0
.../spark/mllib/random/JavaRandomRDDsSuite.java | 0
.../spark/mllib/recommendation/JavaALSSuite.java | 0
.../regression/JavaIsotonicRegressionSuite.java | 0
.../spark/mllib/regression/JavaLassoSuite.java | 0
.../regression/JavaLinearRegressionSuite.java | 0
.../mllib/regression/JavaRidgeRegressionSuite.java | 0
.../JavaStreamingLinearRegressionSuite.java | 0
.../spark/mllib/stat/JavaStatisticsSuite.java | 0
.../spark/mllib/tree/JavaDecisionTreeSuite.java | 0
.../apache/spark/mllib/util/JavaMLUtilsSuite.java | 0
.../org.apache.spark.ml.util.MLFormatRegister | 0
.../date=2018-01/29.5.a_b_EGDP022204.jpg | Bin
.../cls=kittens/date=2018-01/not-image.txt | 0
.../partitioned/cls=kittens/date=2018-02/54893.jpg | Bin
.../cls=kittens/date=2018-02/DP153539.jpg | Bin
.../cls=kittens/date=2018-02/DP802813.jpg | Bin
.../cls=multichannel/date=2018-01/BGRA.png | Bin
.../date=2018-01/BGRA_alpha_60.png | Bin
.../cls=multichannel/date=2018-02/chr30.4.184.jpg | Bin
.../cls=multichannel/date=2018-02/grayscale.jpg | Bin
.../{ => core}/src/test/resources/iris_libsvm.txt | 0
.../src/test/resources/log4j2.properties | 0
.../ml-models/dtc-2.4.7/data/._SUCCESS.crc | Bin
...-406c-894c-ca4eac67c690-c000.snappy.parquet.crc | Bin
.../resources/ml-models/dtc-2.4.7/data/_SUCCESS | 0
...c890-406c-894c-ca4eac67c690-c000.snappy.parquet | Bin
.../ml-models/dtc-2.4.7/metadata/._SUCCESS.crc | Bin
.../ml-models/dtc-2.4.7/metadata/.part-00000.crc | Bin
.../ml-models/dtc-2.4.7/metadata/_SUCCESS | 0
.../ml-models/dtc-2.4.7/metadata/part-00000 | 0
.../ml-models/dtr-2.4.7/data/._SUCCESS.crc | Bin
...-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc | Bin
.../resources/ml-models/dtr-2.4.7/data/_SUCCESS | 0
...a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet | Bin
.../ml-models/dtr-2.4.7/metadata/._SUCCESS.crc | Bin
.../ml-models/dtr-2.4.7/metadata/.part-00000.crc | Bin
.../ml-models/dtr-2.4.7/metadata/_SUCCESS | 0
.../ml-models/dtr-2.4.7/metadata/part-00000 | 0
.../ml-models/gbtc-2.4.7/data/._SUCCESS.crc | Bin
...-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc | Bin
.../resources/ml-models/gbtc-2.4.7/data/_SUCCESS | 0
...c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet | Bin
.../ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc | Bin
.../ml-models/gbtc-2.4.7/metadata/.part-00000.crc | Bin
.../ml-models/gbtc-2.4.7/metadata/_SUCCESS | 0
.../ml-models/gbtc-2.4.7/metadata/part-00000 | 0
.../gbtc-2.4.7/treesMetadata/._SUCCESS.crc | Bin
...-4a90-813c-ddc394101e21-c000.snappy.parquet.crc | Bin
.../ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS | 0
...31e3-4a90-813c-ddc394101e21-c000.snappy.parquet | Bin
.../ml-models/gbtr-2.4.7/data/._SUCCESS.crc | Bin
...-4511-9aab-639288bfae6d-c000.snappy.parquet.crc | Bin
.../resources/ml-models/gbtr-2.4.7/data/_SUCCESS | 0
...d346-4511-9aab-639288bfae6d-c000.snappy.parquet | Bin
.../ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc | Bin
.../ml-models/gbtr-2.4.7/metadata/.part-00000.crc | Bin
.../ml-models/gbtr-2.4.7/metadata/_SUCCESS | 0
.../ml-models/gbtr-2.4.7/metadata/part-00000 | 0
.../gbtr-2.4.7/treesMetadata/._SUCCESS.crc | Bin
...-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc | Bin
.../ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS | 0
...87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet | Bin
.../hashingTF-2.4.4/metadata/._SUCCESS.crc | Bin
.../hashingTF-2.4.4/metadata/.part-00000.crc | Bin
.../ml-models/hashingTF-2.4.4/metadata/_SUCCESS | 0
.../ml-models/hashingTF-2.4.4/metadata/part-00000 | 0
.../ml-models/mlp-2.4.4/data/._SUCCESS.crc | Bin
...-4b90-8231-eb5f6ac12138-c000.snappy.parquet.crc | Bin
.../resources/ml-models/mlp-2.4.4/data/_SUCCESS | 0
...d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet | Bin
.../ml-models/mlp-2.4.4/metadata/._SUCCESS.crc | Bin
.../ml-models/mlp-2.4.4/metadata/.part-00000.crc | Bin
.../ml-models/mlp-2.4.4/metadata/_SUCCESS | 0
.../ml-models/mlp-2.4.4/metadata/part-00000 | 0
.../ml-models/rfc-2.4.7/data/._SUCCESS.crc | Bin
...-4485-b112-25b4b11c9009-c000.snappy.parquet.crc | Bin
.../resources/ml-models/rfc-2.4.7/data/_SUCCESS | 0
...91f8-4485-b112-25b4b11c9009-c000.snappy.parquet | Bin
.../ml-models/rfc-2.4.7/metadata/._SUCCESS.crc | Bin
.../ml-models/rfc-2.4.7/metadata/.part-00000.crc | Bin
.../ml-models/rfc-2.4.7/metadata/_SUCCESS | 0
.../ml-models/rfc-2.4.7/metadata/part-00000 | 0
.../rfc-2.4.7/treesMetadata/._SUCCESS.crc | Bin
...-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc | Bin
.../ml-models/rfc-2.4.7/treesMetadata/_SUCCESS | 0
...b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet | Bin
.../ml-models/rfr-2.4.7/data/._SUCCESS.crc | Bin
...-40fc-b681-981caaeca996-c000.snappy.parquet.crc | Bin
.../resources/ml-models/rfr-2.4.7/data/_SUCCESS | 0
...6edb-40fc-b681-981caaeca996-c000.snappy.parquet | Bin
.../ml-models/rfr-2.4.7/metadata/._SUCCESS.crc | Bin
.../ml-models/rfr-2.4.7/metadata/.part-00000.crc | Bin
.../ml-models/rfr-2.4.7/metadata/_SUCCESS | 0
.../ml-models/rfr-2.4.7/metadata/part-00000 | 0
.../rfr-2.4.7/treesMetadata/._SUCCESS.crc | Bin
...-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc | Bin
.../ml-models/rfr-2.4.7/treesMetadata/_SUCCESS | 0
...d349-447a-9b86-d95edaabcde8-c000.snappy.parquet | Bin
.../strIndexerModel-2.4.4/data/._SUCCESS.crc | Bin
...-4756-b9ca-c5e505dcd898-c000.snappy.parquet.crc | Bin
.../ml-models/strIndexerModel-2.4.4/data/_SUCCESS | 0
...6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet | Bin
.../strIndexerModel-2.4.4/metadata/._SUCCESS.crc | Bin
.../strIndexerModel-2.4.4/metadata/.part-00000.crc | Bin
.../strIndexerModel-2.4.4/metadata/_SUCCESS | 0
.../strIndexerModel-2.4.4/metadata/part-00000 | 0
.../scala/org/apache/spark/ml/FunctionsSuite.scala | 0
.../scala/org/apache/spark/ml/MLEventsSuite.scala | 0
.../scala/org/apache/spark/ml/PipelineSuite.scala | 0
.../scala/org/apache/spark/ml/PredictorSuite.scala | 0
.../scala/org/apache/spark/ml/ann/ANNSuite.scala | 0
.../org/apache/spark/ml/ann/GradientSuite.scala | 0
.../spark/ml/classification/ClassifierSuite.scala | 0
.../DecisionTreeClassifierSuite.scala | 0
.../ml/classification/FMClassifierSuite.scala | 0
.../ml/classification/GBTClassifierSuite.scala | 0
.../spark/ml/classification/LinearSVCSuite.scala | 0
.../classification/LogisticRegressionSuite.scala | 0
.../MultilayerPerceptronClassifierSuite.scala | 0
.../spark/ml/classification/NaiveBayesSuite.scala | 4 +-
.../spark/ml/classification/OneVsRestSuite.scala | 0
.../ProbabilisticClassifierSuite.scala | 0
.../RandomForestClassifierSuite.scala | 0
.../spark/ml/clustering/BisectingKMeansSuite.scala | 0
.../spark/ml/clustering/GaussianMixtureSuite.scala | 0
.../apache/spark/ml/clustering/KMeansSuite.scala | 0
.../org/apache/spark/ml/clustering/LDASuite.scala | 0
.../clustering/PowerIterationClusteringSuite.scala | 0
.../BinaryClassificationEvaluatorSuite.scala | 0
.../ml/evaluation/ClusteringEvaluatorSuite.scala | 0
.../MulticlassClassificationEvaluatorSuite.scala | 0
.../MultilabelClassificationEvaluatorSuite.scala | 0
.../ml/evaluation/RankingEvaluatorSuite.scala | 0
.../ml/evaluation/RegressionEvaluatorSuite.scala | 0
.../apache/spark/ml/feature/BinarizerSuite.scala | 0
.../feature/BucketedRandomProjectionLSHSuite.scala | 0
.../apache/spark/ml/feature/BucketizerSuite.scala | 0
.../spark/ml/feature/ChiSqSelectorSuite.scala | 0
.../spark/ml/feature/CountVectorizerSuite.scala | 0
.../org/apache/spark/ml/feature/DCTSuite.scala | 0
.../spark/ml/feature/ElementwiseProductSuite.scala | 0
.../spark/ml/feature/FeatureHasherSuite.scala | 0
.../apache/spark/ml/feature/HashingTFSuite.scala | 0
.../org/apache/spark/ml/feature/IDFSuite.scala | 0
.../org/apache/spark/ml/feature/ImputerSuite.scala | 0
.../apache/spark/ml/feature/InstanceSuite.scala | 0
.../apache/spark/ml/feature/InteractionSuite.scala | 0
.../org/apache/spark/ml/feature/LSHTest.scala | 0
.../spark/ml/feature/LabeledPointSuite.scala | 0
.../spark/ml/feature/MaxAbsScalerSuite.scala | 0
.../apache/spark/ml/feature/MinHashLSHSuite.scala | 0
.../spark/ml/feature/MinMaxScalerSuite.scala | 0
.../org/apache/spark/ml/feature/NGramSuite.scala | 0
.../apache/spark/ml/feature/NormalizerSuite.scala | 0
.../spark/ml/feature/OneHotEncoderSuite.scala | 0
.../org/apache/spark/ml/feature/PCASuite.scala | 0
.../ml/feature/PolynomialExpansionSuite.scala | 0
.../ml/feature/QuantileDiscretizerSuite.scala | 0
.../spark/ml/feature/RFormulaParserSuite.scala | 0
.../apache/spark/ml/feature/RFormulaSuite.scala | 0
.../spark/ml/feature/RobustScalerSuite.scala | 0
.../spark/ml/feature/SQLTransformerSuite.scala | 0
.../spark/ml/feature/StandardScalerSuite.scala | 0
.../spark/ml/feature/StopWordsRemoverSuite.scala | 0
.../spark/ml/feature/StringIndexerSuite.scala | 0
.../apache/spark/ml/feature/TokenizerSuite.scala | 0
.../feature/UnivariateFeatureSelectorSuite.scala | 0
.../feature/VarianceThresholdSelectorSuite.scala | 0
.../spark/ml/feature/VectorAssemblerSuite.scala | 0
.../spark/ml/feature/VectorIndexerSuite.scala | 0
.../spark/ml/feature/VectorSizeHintSuite.scala | 0
.../spark/ml/feature/VectorSlicerSuite.scala | 0
.../apache/spark/ml/feature/Word2VecSuite.scala | 0
.../org/apache/spark/ml/fpm/FPGrowthSuite.scala | 0
.../org/apache/spark/ml/fpm/PrefixSpanSuite.scala | 0
.../apache/spark/ml/linalg/VectorUDTSuite.scala | 15 --
.../IterativelyReweightedLeastSquaresSuite.scala | 0
.../spark/ml/optim/WeightedLeastSquaresSuite.scala | 0
.../BinaryLogisticBlockAggregatorSuite.scala | 0
.../DifferentiableLossAggregatorSuite.scala | 0
.../aggregator/HingeBlockAggregatorSuite.scala | 0
.../aggregator/HuberBlockAggregatorSuite.scala | 0
.../LeastSquaresBlockAggregatorSuite.scala | 0
.../MultinomialLogisticBlockAggregatorSuite.scala | 0
.../loss/DifferentiableRegularizationSuite.scala | 0
.../spark/ml/optim/loss/RDDLossFunctionSuite.scala | 0
.../org/apache/spark/ml/param/ParamsSuite.scala | 101 +++++++++++++
.../org/apache/spark/ml/python/MLSerDeSuite.scala | 0
.../org/apache/spark/ml/r/RWrapperUtilsSuite.scala | 2 +-
.../apache/spark/ml/recommendation/ALSSuite.scala | 0
.../spark/ml/recommendation/CollectTopKSuite.scala | 0
.../ml/regression/AFTSurvivalRegressionSuite.scala | 0
.../ml/regression/DecisionTreeRegressorSuite.scala | 0
.../spark/ml/regression/FMRegressorSuite.scala | 0
.../spark/ml/regression/GBTRegressorSuite.scala | 0
.../GeneralizedLinearRegressionSuite.scala | 0
.../ml/regression/IsotonicRegressionSuite.scala | 0
.../ml/regression/LinearRegressionSuite.scala | 0
.../ml/regression/RandomForestRegressorSuite.scala | 0
.../ml/source/image/ImageFileFormatSuite.scala | 0
.../ml/source/libsvm/LibSVMRelationSuite.scala | 0
.../org/apache/spark/ml/stat/ANOVATestSuite.scala | 0
.../apache/spark/ml/stat/ChiSquareTestSuite.scala | 0
.../apache/spark/ml/stat/CorrelationSuite.scala | 0
.../org/apache/spark/ml/stat/FValueTestSuite.scala | 0
.../spark/ml/stat/KolmogorovSmirnovTestSuite.scala | 0
.../spark/ml/stat/MultiClassSummarizerSuite.scala | 0
.../org/apache/spark/ml/stat/SummarizerSuite.scala | 0
.../spark/ml/tree/impl/BaggedPointSuite.scala | 0
.../ml/tree/impl/GradientBoostedTreesSuite.scala | 0
.../spark/ml/tree/impl/RandomForestSuite.scala | 0
.../apache/spark/ml/tree/impl/TreePointSuite.scala | 0
.../org/apache/spark/ml/tree/impl/TreeTests.scala | 0
.../spark/ml/tuning/CrossValidatorSuite.scala | 0
.../spark/ml/tuning/ParamGridBuilderSuite.scala | 0
.../ml/tuning/TrainValidationSplitSuite.scala | 0
.../ml/tuning/ValidatorParamsSuiteHelpers.scala | 0
.../spark/ml/util/DefaultReadWriteTest.scala | 0
.../scala/org/apache/spark/ml/util/MLTest.scala | 0
.../org/apache/spark/ml/util/MLTestSuite.scala | 0
.../org/apache/spark/ml/util/MLTestingUtils.scala | 0
.../apache/spark/ml/util/PMMLReadWriteTest.scala | 0
.../scala/org/apache/spark/ml/util/PMMLUtils.scala | 0
.../org/apache/spark/ml/util/ReadWriteSuite.scala | 0
.../org/apache/spark/ml/util/StopwatchSuite.scala | 0
.../org/apache/spark/ml/util/TempDirectory.scala | 0
.../mllib/api/python/PythonMLLibAPISuite.scala | 0
.../classification/LogisticRegressionSuite.scala | 0
.../mllib/classification/NaiveBayesSuite.scala | 0
.../spark/mllib/classification/SVMSuite.scala | 0
.../StreamingLogisticRegressionSuite.scala | 0
.../mllib/clustering/BisectingKMeansSuite.scala | 0
.../mllib/clustering/DistanceMeasureSuite.scala | 0
.../mllib/clustering/GaussianMixtureSuite.scala | 0
.../spark/mllib/clustering/KMeansSuite.scala | 0
.../apache/spark/mllib/clustering/LDASuite.scala | 0
.../clustering/PowerIterationClusteringSuite.scala | 0
.../mllib/clustering/StreamingKMeansSuite.scala | 0
.../mllib/evaluation/AreaUnderCurveSuite.scala | 0
.../BinaryClassificationMetricsSuite.scala | 0
.../mllib/evaluation/MulticlassMetricsSuite.scala | 0
.../mllib/evaluation/MultilabelMetricsSuite.scala | 0
.../mllib/evaluation/RankingMetricsSuite.scala | 0
.../mllib/evaluation/RegressionMetricsSuite.scala | 0
.../spark/mllib/feature/ChiSqSelectorSuite.scala | 0
.../mllib/feature/ElementwiseProductSuite.scala | 0
.../spark/mllib/feature/HashingTFSuite.scala | 0
.../org/apache/spark/mllib/feature/IDFSuite.scala | 0
.../spark/mllib/feature/NormalizerSuite.scala | 0
.../org/apache/spark/mllib/feature/PCASuite.scala | 0
.../spark/mllib/feature/StandardScalerSuite.scala | 0
.../apache/spark/mllib/feature/Word2VecSuite.scala | 0
.../spark/mllib/fpm/AssociationRulesSuite.scala | 0
.../org/apache/spark/mllib/fpm/FPGrowthSuite.scala | 0
.../org/apache/spark/mllib/fpm/FPTreeSuite.scala | 0
.../apache/spark/mllib/fpm/PrefixSpanSuite.scala | 0
.../org/apache/spark/mllib/linalg/BLASSuite.scala | 0
.../mllib/linalg/BreezeMatrixConversionSuite.scala | 0
.../mllib/linalg/BreezeVectorConversionSuite.scala | 0
.../apache/spark/mllib/linalg/MatricesSuite.scala | 0
.../mllib/linalg/UDTSerializationBenchmark.scala | 0
.../apache/spark/mllib/linalg/VectorsSuite.scala | 0
.../linalg/distributed/BlockMatrixSuite.scala | 0
.../linalg/distributed/CoordinateMatrixSuite.scala | 0
.../linalg/distributed/IndexedRowMatrixSuite.scala | 0
.../mllib/linalg/distributed/RowMatrixSuite.scala | 0
.../mllib/optimization/GradientDescentSuite.scala | 0
.../spark/mllib/optimization/LBFGSSuite.scala | 0
.../spark/mllib/optimization/NNLSSuite.scala | 0
.../BinaryClassificationPMMLModelExportSuite.scala | 0
.../GeneralizedLinearPMMLModelExportSuite.scala | 0
.../pmml/export/KMeansPMMLModelExportSuite.scala | 0
.../pmml/export/PMMLModelExportFactorySuite.scala | 0
.../mllib/random/RandomDataGeneratorSuite.scala | 0
.../spark/mllib/random/RandomRDDsSuite.scala | 0
.../spark/mllib/rdd/MLPairRDDFunctionsSuite.scala | 0
.../apache/spark/mllib/rdd/RDDFunctionsSuite.scala | 0
.../spark/mllib/recommendation/ALSSuite.scala | 0
.../MatrixFactorizationModelSuite.scala | 0
.../mllib/regression/IsotonicRegressionSuite.scala | 0
.../spark/mllib/regression/LabeledPointSuite.scala | 0
.../apache/spark/mllib/regression/LassoSuite.scala | 0
.../mllib/regression/LinearRegressionSuite.scala | 0
.../mllib/regression/RidgeRegressionSuite.scala | 0
.../StreamingLinearRegressionSuite.scala | 0
.../apache/spark/mllib/stat/CorrelationSuite.scala | 0
.../spark/mllib/stat/HypothesisTestSuite.scala | 0
.../spark/mllib/stat/KernelDensitySuite.scala | 0
.../stat/MultivariateOnlineSummarizerSuite.scala | 0
.../spark/mllib/stat/StreamingTestSuite.scala | 0
.../distribution/MultivariateGaussianSuite.scala | 0
.../spark/mllib/tree/DecisionTreeSuite.scala | 0
.../spark/mllib/tree/EnsembleTestHelper.scala | 0
.../mllib/tree/GradientBoostedTreesSuite.scala | 0
.../apache/spark/mllib/tree/ImpuritySuite.scala | 0
.../spark/mllib/tree/RandomForestSuite.scala | 0
.../mllib/util/LocalClusterSparkContext.scala | 0
.../org/apache/spark/mllib/util/MLUtilsSuite.scala | 0
.../spark/mllib/util/MLlibTestSparkContext.scala | 0
.../spark/mllib/util/NumericParserSuite.scala | 0
.../org/apache/spark/mllib/util/TestingUtils.scala | 0
.../spark/mllib/util/TestingUtilsSuite.scala | 0
.../benchmarks/BLASBenchmark-jdk11-results.txt | 0
.../benchmarks/BLASBenchmark-jdk17-results.txt | 0
.../local}/benchmarks/BLASBenchmark-results.txt | 0
{mllib-local => mllib/local}/pom.xml | 2 +-
.../scala/org/apache/spark/ml/impl/Utils.scala | 0
.../scala/org/apache/spark/ml/linalg/BLAS.scala | 0
.../org/apache/spark/ml/linalg/Matrices.scala | 0
.../scala/org/apache/spark/ml/linalg/Vectors.scala | 0
.../stat/distribution/MultivariateGaussian.scala | 0
.../org/apache/spark/ml/SparkMLFunSuite.scala | 0
.../org/apache/spark/ml/impl/UtilsSuite.scala | 0
.../org/apache/spark/ml/linalg/BLASBenchmark.scala | 0
.../org/apache/spark/ml/linalg/BLASSuite.scala | 0
.../ml/linalg/BreezeMatrixConversionSuite.scala | 0
.../ml/linalg/BreezeVectorConversionSuite.scala | 0
.../org/apache/spark/ml/linalg/MatricesSuite.scala | 0
.../org/apache/spark/ml/linalg/VectorsSuite.scala | 0
.../distribution/MultivariateGaussianSuite.scala | 0
.../org/apache/spark/ml/util/TestingUtils.scala | 0
.../apache/spark/ml/util/TestingUtilsSuite.scala | 0
pom.xml | 5 +-
project/MimaExcludes.scala | 83 +++++++++++
project/SparkBuild.scala | 7 +-
762 files changed, 994 insertions(+), 382 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index ebc40e92791..9f96f5beab3 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -149,7 +149,7 @@ jobs:
catalyst, hive-thriftserver
- >-
streaming, sql-kafka-0-10, streaming-kafka-0-10,
- mllib-local, mllib,
+ mllib-local, mllib-common, mllib,
yarn, mesos, kubernetes, hadoop-cloud, spark-ganglia-lgpl,
connect, protobuf
# Here, we split Hive and SQL tests into some of slow ones and the rest of them.
diff --git a/connector/connect/client/jvm/pom.xml b/connector/connect/client/jvm/pom.xml
index 7606795f820..ac4b1655f5e 100644
--- a/connector/connect/client/jvm/pom.xml
+++ b/connector/connect/client/jvm/pom.xml
@@ -62,6 +62,18 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-mllib-common_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Estimator.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Estimator.scala
similarity index 63%
copy from mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
copy to connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Estimator.scala
index 3a02e2be6fe..144a10641c7 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Estimator.scala
@@ -31,15 +31,22 @@ abstract class Estimator[M <: Model[M]] extends PipelineStage {
/**
* Fits a single model to the input data with optional parameters.
*
- * @param dataset input dataset
- * @param firstParamPair the first param pair, overrides embedded params
- * @param otherParamPairs other param pairs. These values override any specified in this
- * Estimator's embedded ParamMap.
- * @return fitted model
+ * @param dataset
+ * input dataset
+ * @param firstParamPair
+ * the first param pair, overrides embedded params
+ * @param otherParamPairs
+ * other param pairs. These values override any specified in this Estimator's embedded
+ * ParamMap.
+ * @return
+ * fitted model
*/
- @Since("2.0.0")
+ @Since("3.5.0")
@varargs
- def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): M = {
+ def fit(
+ dataset: Dataset[_],
+ firstParamPair: ParamPair[_],
+ otherParamPairs: ParamPair[_]*): M = {
val map = new ParamMap()
.put(firstParamPair)
.put(otherParamPairs: _*)
@@ -49,12 +56,14 @@ abstract class Estimator[M <: Model[M]] extends PipelineStage {
/**
* Fits a single model to the input data with provided parameter map.
*
- * @param dataset input dataset
- * @param paramMap Parameter map.
- * These values override any specified in this Estimator's embedded ParamMap.
- * @return fitted model
+ * @param dataset
+ * input dataset
+ * @param paramMap
+ * Parameter map. These values override any specified in this Estimator's embedded ParamMap.
+ * @return
+ * fitted model
*/
- @Since("2.0.0")
+ @Since("3.5.0")
def fit(dataset: Dataset[_], paramMap: ParamMap): M = {
copy(paramMap).fit(dataset)
}
@@ -62,23 +71,27 @@ abstract class Estimator[M <: Model[M]] extends PipelineStage {
/**
* Fits a model to the input data.
*/
- @Since("2.0.0")
+ @Since("3.5.0")
def fit(dataset: Dataset[_]): M
/**
- * Fits multiple models to the input data with multiple sets of parameters.
- * The default implementation uses a for loop on each parameter map.
- * Subclasses could override this to optimize multi-model training.
+ * Fits multiple models to the input data with multiple sets of parameters. The default
+ * implementation uses a for loop on each parameter map. Subclasses could override this to
+ * optimize multi-model training.
*
- * @param dataset input dataset
- * @param paramMaps An array of parameter maps.
- * These values override any specified in this Estimator's embedded ParamMap.
- * @return fitted models, matching the input parameter maps
+ * @param dataset
+ * input dataset
+ * @param paramMaps
+ * An array of parameter maps. These values override any specified in this Estimator's
+ * embedded ParamMap.
+ * @return
+ * fitted models, matching the input parameter maps
*/
- @Since("2.0.0")
+ @Since("3.5.0")
def fit(dataset: Dataset[_], paramMaps: Seq[ParamMap]): Seq[M] = {
paramMaps.map(fit(dataset, _))
}
+ @Since("3.5.0")
override def copy(extra: ParamMap): Estimator[M]
}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Model.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Model.scala
similarity index 87%
copy from mllib/src/main/scala/org/apache/spark/ml/Model.scala
copy to connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Model.scala
index 709a1124cbb..a5d6aa1a079 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Model.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Model.scala
@@ -17,30 +17,37 @@
package org.apache.spark.ml
+import org.apache.spark.annotation.Since
import org.apache.spark.ml.param.ParamMap
/**
* A fitted model, i.e., a [[Transformer]] produced by an [[Estimator]].
*
- * @tparam M model type
+ * @tparam M
+ * model type
*/
abstract class Model[M <: Model[M]] extends Transformer {
+
/**
* The parent estimator that produced this model.
- * @note For ensembles' component Models, this value can be null.
+ * @note
+ * For ensembles' component Models, this value can be null.
*/
@transient var parent: Estimator[M] = _
/**
* Sets the parent of this model (Java API).
*/
+ @Since("3.5.0")
def setParent(parent: Estimator[M]): M = {
this.parent = parent
this.asInstanceOf[M]
}
/** Indicates whether this [[Model]] has a corresponding parent. */
+ @Since("3.5.0")
def hasParent: Boolean = parent != null
+ @Since("3.5.0")
override def copy(extra: ParamMap): M
}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Pipeline.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Pipeline.scala
new file mode 100644
index 00000000000..cebbcd167ce
--- /dev/null
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Pipeline.scala
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.param.{ParamMap, Params}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A stage in a pipeline, either an [[Estimator]] or a [[Transformer]].
+ */
+abstract class PipelineStage extends Params with Logging {
+
+ /**
+ * Check transform validity and derive the output schema from the input schema.
+ *
+ * We check validity for interactions between parameters during `transformSchema` and raise an
+ * exception if any parameter value is invalid. Parameter value checks which do not depend on
+ * other parameters are handled by `Param.validate()`.
+ *
+ * Typical implementation should first conduct verification on schema change and parameter
+ * validity, including complex parameter interaction checks.
+ */
+ def transformSchema(schema: StructType): StructType
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Derives the output schema from the input schema and parameters, optionally with logging.
+ *
+ * This should be optimistic. If it is unclear whether the schema will be valid, then it should
+ * be assumed valid until proven otherwise.
+ */
+ @DeveloperApi
+ protected def transformSchema(schema: StructType, logging: Boolean): StructType = {
+ if (logging) {
+ logDebug(s"Input schema: ${schema.json}")
+ }
+ val outputSchema = transformSchema(schema)
+ if (logging) {
+ logDebug(s"Expected output schema: ${outputSchema.json}")
+ }
+ outputSchema
+ }
+
+ override def copy(extra: ParamMap): PipelineStage
+}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Predictor.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Predictor.scala
new file mode 100644
index 00000000000..517d5e060f5
--- /dev/null
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Predictor.scala
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.linalg.VectorUDT
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.util.SchemaUtils
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.types.{DataType, StructType}
+
+/**
+ * Abstraction for prediction problems (regression and classification). It accepts all NumericType
+ * labels and will automatically cast it to DoubleType in `fit()`. If this predictor supports
+ * weights, it accepts all NumericType weights, which will be automatically casted to DoubleType
+ * in `fit()`.
+ *
+ * @tparam FeaturesType
+ * Type of features. E.g., `VectorUDT` for vector features.
+ * @tparam Learner
+ * Specialization of this class. If you subclass this type, use this type parameter to specify
+ * the concrete type.
+ * @tparam M
+ * Specialization of [[PredictionModel]]. If you subclass this type, use this type parameter to
+ * specify the concrete type for the corresponding model.
+ */
+abstract class Predictor[
+ FeaturesType,
+ Learner <: Predictor[FeaturesType, Learner, M],
+ M <: PredictionModel[FeaturesType, M]]
+ extends Estimator[M]
+ with PredictorParams {
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setLabelCol(value: String): Learner = set(labelCol, value).asInstanceOf[Learner]
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setFeaturesCol(value: String): Learner = set(featuresCol, value).asInstanceOf[Learner]
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setPredictionCol(value: String): Learner = set(predictionCol, value).asInstanceOf[Learner]
+
+ @Since("3.5.0")
+ override def fit(dataset: Dataset[_]): M = {
+ // TODO: should send the id of the input dataset and the latest params to the server,
+ // then invoke the 'fit' method of the remote predictor
+ throw new NotImplementedError
+ }
+
+ @Since("3.5.0")
+ override def copy(extra: ParamMap): Learner
+
+ /**
+ * Returns the SQL DataType corresponding to the FeaturesType type parameter.
+ *
+ * This is used by `validateAndTransformSchema()`. This workaround is needed since SQL has
+ * different APIs for Scala and Java.
+ *
+ * The default value is VectorUDT, but it may be overridden if FeaturesType is not Vector.
+ */
+ private[ml] def featuresDataType: DataType = new VectorUDT
+
+ override def transformSchema(schema: StructType): StructType = {
+ validateAndTransformSchema(schema, fitting = true, featuresDataType)
+ }
+}
+
+/**
+ * Abstraction for a model for prediction tasks (regression and classification).
+ *
+ * @tparam FeaturesType
+ * Type of features. E.g., `VectorUDT` for vector features.
+ * @tparam M
+ * Specialization of [[PredictionModel]]. If you subclass this type, use this type parameter to
+ * specify the concrete type for the corresponding model.
+ */
+abstract class PredictionModel[FeaturesType, M <: PredictionModel[FeaturesType, M]]
+ extends Model[M]
+ with PredictorParams {
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setFeaturesCol(value: String): M = set(featuresCol, value).asInstanceOf[M]
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setPredictionCol(value: String): M = set(predictionCol, value).asInstanceOf[M]
+
+ /** Returns the number of features the model was trained on. If unknown, returns -1 */
+ @Since("3.5.0")
+ def numFeatures: Int = -1
+
+ /**
+ * Returns the SQL DataType corresponding to the FeaturesType type parameter.
+ *
+ * This is used by `validateAndTransformSchema()`. This workaround is needed since SQL has
+ * different APIs for Scala and Java.
+ *
+ * The default value is VectorUDT, but it may be overridden if FeaturesType is not Vector.
+ */
+ protected def featuresDataType: DataType = new VectorUDT
+
+ @Since("3.5.0")
+ override def transformSchema(schema: StructType): StructType = {
+ var outputSchema = validateAndTransformSchema(schema, fitting = false, featuresDataType)
+ if ($(predictionCol).nonEmpty) {
+ outputSchema = SchemaUtils.updateNumeric(outputSchema, $(predictionCol))
+ }
+ outputSchema
+ }
+
+ /**
+ * Transforms dataset by reading from [[featuresCol]], calling `predict`, and storing the
+ * predictions as a new column [[predictionCol]].
+ *
+ * @param dataset
+ * input dataset
+ * @return
+ * transformed dataset with [[predictionCol]] of type `Double`
+ */
+ @Since("3.5.0")
+ override def transform(dataset: Dataset[_]): DataFrame = {
+ transformSchema(dataset.schema, logging = true)
+ if ($(predictionCol).nonEmpty) {
+ transformImpl(dataset)
+ } else {
+ this.logWarning(
+ s"$uid: Predictor.transform() does nothing" +
+ " because no output columns were set.")
+ dataset.toDF
+ }
+ }
+
+ protected def transformImpl(dataset: Dataset[_]): DataFrame = {
+ // TODO: should send the id of the input dataset and the latest params to the server,
+ // then invoke the 'transform' method of the remote model
+ throw new NotImplementedError
+ }
+
+ /**
+ * Predict label for the given features. This method is used to implement `transform()` and
+ * output [[predictionCol]].
+ */
+ @Since("3.5.0")
+ def predict(features: FeaturesType): Double
+}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Transformer.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Transformer.scala
similarity index 78%
copy from mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
copy to connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Transformer.scala
index 3b60b5ae294..4eebf031b90 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/Transformer.scala
@@ -25,7 +25,6 @@ import org.apache.spark.internal.Logging
import org.apache.spark.ml.param._
import org.apache.spark.ml.param.shared._
import org.apache.spark.sql.{DataFrame, Dataset}
-import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
/**
@@ -35,12 +34,16 @@ abstract class Transformer extends PipelineStage {
/**
* Transforms the dataset with optional parameters
- * @param dataset input dataset
- * @param firstParamPair the first param pair, overwrite embedded params
- * @param otherParamPairs other param pairs, overwrite embedded params
- * @return transformed dataset
+ * @param dataset
+ * input dataset
+ * @param firstParamPair
+ * the first param pair, overwrite embedded params
+ * @param otherParamPairs
+ * other param pairs, overwrite embedded params
+ * @return
+ * transformed dataset
*/
- @Since("2.0.0")
+ @Since("3.5.0")
@varargs
def transform(
dataset: Dataset[_],
@@ -54,11 +57,14 @@ abstract class Transformer extends PipelineStage {
/**
* Transforms the dataset with provided parameter map as additional parameters.
- * @param dataset input dataset
- * @param paramMap additional parameters, overwrite embedded params
- * @return transformed dataset
+ * @param dataset
+ * input dataset
+ * @param paramMap
+ * additional parameters, overwrite embedded params
+ * @return
+ * transformed dataset
*/
- @Since("2.0.0")
+ @Since("3.5.0")
def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame = {
this.copy(paramMap).transform(dataset)
}
@@ -66,35 +72,42 @@ abstract class Transformer extends PipelineStage {
/**
* Transforms the input dataset.
*/
- @Since("2.0.0")
+ @Since("3.5.0")
def transform(dataset: Dataset[_]): DataFrame
+ @Since("3.5.0")
override def copy(extra: ParamMap): Transformer
}
/**
- * Abstract class for transformers that take one input column, apply transformation, and output the
- * result as a new column.
+ * Abstract class for transformers that take one input column, apply transformation, and output
+ * the result as a new column.
*/
abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
- extends Transformer with HasInputCol with HasOutputCol with Logging {
+ extends Transformer
+ with HasInputCol
+ with HasOutputCol
+ with Logging {
/** @group setParam */
+ @Since("3.5.0")
def setInputCol(value: String): T = set(inputCol, value).asInstanceOf[T]
/** @group setParam */
+ @Since("3.5.0")
def setOutputCol(value: String): T = set(outputCol, value).asInstanceOf[T]
/**
* Creates the transform function using the given param map. The input param map already takes
- * account of the embedded param map. So the param values should be determined solely by the input
- * param map.
+ * account of the embedded param map. So the param values should be determined solely by the
+ * input param map.
*/
protected def createTransformFunc: IN => OUT
/**
* Returns the data type of the output column.
*/
+ @Since("3.5.0")
protected def outputDataType: DataType
/**
@@ -102,6 +115,7 @@ abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer
*/
protected def validateInputType(inputType: DataType): Unit = {}
+ @Since("3.5.0")
override def transformSchema(schema: StructType): StructType = {
val inputType = schema($(inputCol)).dataType
validateInputType(inputType)
@@ -114,11 +128,11 @@ abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer
}
override def transform(dataset: Dataset[_]): DataFrame = {
- val outputSchema = transformSchema(dataset.schema, logging = true)
- val transformUDF = udf(this.createTransformFunc)
- dataset.withColumn($(outputCol), transformUDF(dataset($(inputCol))),
- outputSchema($(outputCol)).metadata)
+ // TODO: should send the id of the input dataset and the latest params to the server,
+ // then invoke the 'transform' method of the remote model
+ throw new NotImplementedError
}
+ @Since("3.5.0")
override def copy(extra: ParamMap): T = defaultCopy(extra)
}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/classification/Classifier.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
new file mode 100644
index 00000000000..9adf49866b4
--- /dev/null
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.classification
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{PredictionModel, Predictor}
+import org.apache.spark.ml.linalg.Vector
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Single-label binary or multiclass classification. Classes are indexed {0, 1, ..., numClasses -
+ * 1}.
+ *
+ * @tparam FeaturesType
+ * Type of input features. E.g., `Vector`
+ * @tparam E
+ * Concrete Estimator type
+ * @tparam M
+ * Concrete Model type
+ */
+abstract class Classifier[
+ FeaturesType,
+ E <: Classifier[FeaturesType, E, M],
+ M <: ClassificationModel[FeaturesType, M]]
+ extends Predictor[FeaturesType, E, M]
+ with ClassifierParams {
+
+ @Since("3.5.0")
+ def setRawPredictionCol(value: String): E = set(rawPredictionCol, value).asInstanceOf[E]
+
+ // TODO: defaultEvaluator (follow-up PR)
+}
+
+/**
+ * Model produced by a [[Classifier]]. Classes are indexed {0, 1, ..., numClasses - 1}.
+ *
+ * @tparam FeaturesType
+ * Type of input features. E.g., `Vector`
+ * @tparam M
+ * Concrete Model type
+ */
+abstract class ClassificationModel[FeaturesType, M <: ClassificationModel[FeaturesType, M]]
+ extends PredictionModel[FeaturesType, M]
+ with ClassifierParams {
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setRawPredictionCol(value: String): M = set(rawPredictionCol, value).asInstanceOf[M]
+
+ /** Number of classes (values which the label can take). */
+ @Since("3.5.0")
+ def numClasses: Int
+
+ @Since("3.5.0")
+ override def transformSchema(schema: StructType): StructType = {
+ var outputSchema = super.transformSchema(schema)
+ if ($(predictionCol).nonEmpty) {
+ outputSchema = SchemaUtils.updateNumValues(schema, $(predictionCol), numClasses)
+ }
+ if ($(rawPredictionCol).nonEmpty) {
+ outputSchema =
+ SchemaUtils.updateAttributeGroupSize(outputSchema, $(rawPredictionCol), numClasses)
+ }
+ outputSchema
+ }
+
+ /**
+ * Transforms dataset by reading from [[featuresCol]], and appending new columns as specified by
+ * parameters:
+ * - predicted labels as [[predictionCol]] of type `Double`
+ * - raw predictions (confidences) as [[rawPredictionCol]] of type `Vector`.
+ *
+ * @param dataset
+ * input dataset
+ * @return
+ * transformed dataset
+ */
+ @Since("3.5.0")
+ override def transform(dataset: Dataset[_]): DataFrame = {
+ // TODO: should send the id of the input dataset and the latest params to the server,
+ // then invoke the 'transform' method of the remote model
+ throw new NotImplementedError
+ }
+
+ final override def transformImpl(dataset: Dataset[_]): DataFrame =
+ throw new UnsupportedOperationException(s"transformImpl is not supported in $getClass")
+
+ /**
+ * Predict label for the given features. This method is used to implement `transform()` and
+ * output [[predictionCol]].
+ *
+ * This default implementation for classification predicts the index of the maximum value from
+ * `predictRaw()`.
+ */
+ @Since("3.5.0")
+ override def predict(features: FeaturesType): Double = {
+ // TODO: should send the vector to the server,
+ // then invoke the 'predict' method of the remote model
+
+ // Note: Subclass may need to override this, since the result
+ // maybe adjusted by param like `thresholds`.
+ throw new NotImplementedError
+ }
+
+ /**
+ * Raw prediction for each possible label. The meaning of a "raw" prediction may vary between
+ * algorithms, but it intuitively gives a measure of confidence in each possible label (where
+ * larger = more confident). This internal method is used to implement `transform()` and output
+ * [[rawPredictionCol]].
+ *
+ * @return
+ * vector where element i is the raw prediction for label i. This raw prediction may be any
+ * real number, where a larger value indicates greater confidence for that label.
+ */
+ @Since("3.5.0")
+ def predictRaw(features: FeaturesType): Vector = {
+ // TODO: should send the vector to the server,
+ // then invoke the 'predictRaw' method of the remote model
+ throw new NotImplementedError
+ }
+}
diff --git a/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
new file mode 100644
index 00000000000..e4db8a047fa
--- /dev/null
+++ b/connector/connect/client/jvm/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.classification
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.linalg.Vector
+import org.apache.spark.ml.util.SchemaUtils
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Single-label binary or multiclass classifier which can output class conditional probabilities.
+ *
+ * @tparam FeaturesType
+ * Type of input features. E.g., `Vector`
+ * @tparam E
+ * Concrete Estimator type
+ * @tparam M
+ * Concrete Model type
+ */
+abstract class ProbabilisticClassifier[
+ FeaturesType,
+ E <: ProbabilisticClassifier[FeaturesType, E, M],
+ M <: ProbabilisticClassificationModel[FeaturesType, M]]
+ extends Classifier[FeaturesType, E, M]
+ with ProbabilisticClassifierParams {
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setProbabilityCol(value: String): E = set(probabilityCol, value).asInstanceOf[E]
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setThresholds(value: Array[Double]): E = set(thresholds, value).asInstanceOf[E]
+}
+
+/**
+ * Model produced by a [[ProbabilisticClassifier]]. Classes are indexed {0, 1, ..., numClasses -
+ * 1}.
+ *
+ * @tparam FeaturesType
+ * Type of input features. E.g., `Vector`
+ * @tparam M
+ * Concrete Model type
+ */
+abstract class ProbabilisticClassificationModel[
+ FeaturesType,
+ M <: ProbabilisticClassificationModel[FeaturesType, M]]
+ extends ClassificationModel[FeaturesType, M]
+ with ProbabilisticClassifierParams {
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setProbabilityCol(value: String): M = set(probabilityCol, value).asInstanceOf[M]
+
+ /** @group setParam */
+ @Since("3.5.0")
+ def setThresholds(value: Array[Double]): M = {
+ require(
+ value.length == numClasses,
+ this.getClass.getSimpleName +
+ ".setThresholds() called with non-matching numClasses and thresholds.length." +
+ s" numClasses=$numClasses, but thresholds has length ${value.length}")
+ set(thresholds, value).asInstanceOf[M]
+ }
+
+ @Since("3.5.0")
+ override def transformSchema(schema: StructType): StructType = {
+ var outputSchema = super.transformSchema(schema)
+ if ($(probabilityCol).nonEmpty) {
+ outputSchema =
+ SchemaUtils.updateAttributeGroupSize(outputSchema, $(probabilityCol), numClasses)
+ }
+ outputSchema
+ }
+
+ /**
+ * Transforms dataset by reading from [[featuresCol]], and appending new columns as specified by
+ * parameters:
+ * - predicted labels as [[predictionCol]] of type `Double`
+ * - raw predictions (confidences) as [[rawPredictionCol]] of type `Vector`
+ * - probability of each class as [[probabilityCol]] of type `Vector`.
+ *
+ * @param dataset
+ * input dataset
+ * @return
+ * transformed dataset
+ */
+ @Since("3.5.0")
+ override def transform(dataset: Dataset[_]): DataFrame = {
+ // TODO: should send the id of the input dataset and the latest params to the server,
+ // then invoke the 'transform' method of the remote model
+ throw new NotImplementedError
+ }
+
+ /**
+ * Predict the probability of each class given the features. These predictions are also called
+ * class conditional probabilities.
+ *
+ * This internal method is used to implement `transform()` and output [[probabilityCol]].
+ *
+ * @return
+ * Estimated class conditional probabilities
+ */
+ @Since("3.5.0")
+ def predictProbability(features: FeaturesType): Vector = {
+ // TODO: should send the vector to the server,
+ // then invoke the 'predictProbability' method of the remote model
+ throw new NotImplementedError
+ }
+}
diff --git a/connector/connect/server/pom.xml b/connector/connect/server/pom.xml
index 079d07db362..4d8e082a2db 100644
--- a/connector/connect/server/pom.xml
+++ b/connector/connect/server/pom.xml
@@ -93,6 +93,18 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-mllib_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 751f0687f2c..5fcc4365d20 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -271,9 +271,32 @@ sql_kafka = Module(
],
)
+mllib_local = Module(
+ name="mllib-local",
+ dependencies=[tags, core],
+ source_file_regexes=[
+ "mllib/local",
+ ],
+ sbt_test_goals=[
+ "mllib-local/test",
+ ],
+)
+
+
+mllib_common = Module(
+ name="mllib-common",
+ dependencies=[tags, mllib_local, sql],
+ source_file_regexes=[
+ "mllib/common",
+ ],
+ sbt_test_goals=[
+ "mllib-common/test",
+ ],
+)
+
connect = Module(
name="connect",
- dependencies=[hive],
+ dependencies=[hive, mllib_common],
source_file_regexes=[
"connector/connect",
],
@@ -358,24 +381,12 @@ streaming_kafka_0_10 = Module(
)
-mllib_local = Module(
- name="mllib-local",
- dependencies=[tags, core],
- source_file_regexes=[
- "mllib-local",
- ],
- sbt_test_goals=[
- "mllib-local/test",
- ],
-)
-
-
mllib = Module(
name="mllib",
- dependencies=[mllib_local, streaming, sql],
+ dependencies=[mllib_local, mllib_common, streaming, sql],
source_file_regexes=[
"data/mllib/",
- "mllib/",
+ "mllib/core/",
],
sbt_test_goals=[
"mllib/test",
diff --git a/dev/sparktestsupport/utils.py b/dev/sparktestsupport/utils.py
index 6b190eb5ab2..5c270d0948e 100755
--- a/dev/sparktestsupport/utils.py
+++ b/dev/sparktestsupport/utils.py
@@ -112,22 +112,25 @@ def determine_modules_to_test(changed_modules, deduplicated=True):
>>> sorted([x.name for x in determine_modules_to_test([modules.sql])])
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'connect', 'docker-integration-tests', 'examples', 'hive', 'hive-thriftserver',
- 'mllib', 'protobuf', 'pyspark-connect', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas',
- 'pyspark-pandas-slow', 'pyspark-sql', 'repl', 'sparkr', 'sql', 'sql-kafka-0-10']
+ 'mllib', 'mllib-common', 'protobuf', 'pyspark-connect', 'pyspark-ml', 'pyspark-mllib',
+ 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-sql', 'repl', 'sparkr', 'sql',
+ 'sql-kafka-0-10']
>>> sorted([x.name for x in determine_modules_to_test(
... [modules.sparkr, modules.sql], deduplicated=False)])
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'connect', 'docker-integration-tests', 'examples', 'hive', 'hive-thriftserver',
- 'mllib', 'protobuf', 'pyspark-connect', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas',
- 'pyspark-pandas-slow', 'pyspark-sql', 'repl', 'sparkr', 'sql', 'sql-kafka-0-10']
+ 'mllib', 'mllib-common', 'protobuf', 'pyspark-connect', 'pyspark-ml', 'pyspark-mllib',
+ 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-sql', 'repl', 'sparkr', 'sql',
+ 'sql-kafka-0-10']
>>> sorted([x.name for x in determine_modules_to_test(
... [modules.sql, modules.core], deduplicated=False)])
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'catalyst', 'connect', 'core', 'docker-integration-tests', 'examples', 'graphx',
- 'hive', 'hive-thriftserver', 'mllib', 'mllib-local', 'protobuf', 'pyspark-connect',
- 'pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-slow',
- 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming', 'repl', 'root', 'sparkr', 'sql',
- 'sql-kafka-0-10', 'streaming', 'streaming-kafka-0-10', 'streaming-kinesis-asl']
+ 'hive', 'hive-thriftserver', 'mllib', 'mllib-common', 'mllib-local', 'protobuf',
+ 'pyspark-connect', 'pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas',
+ 'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming', 'repl',
+ 'root', 'sparkr', 'sql', 'sql-kafka-0-10', 'streaming', 'streaming-kafka-0-10',
+ 'streaming-kinesis-asl']
"""
modules_to_test = set()
for module in changed_modules:
diff --git a/mllib-local/pom.xml b/mllib/common/pom.xml
similarity index 71%
copy from mllib-local/pom.xml
copy to mllib/common/pom.xml
index 00c16a8b6a5..41275b24499 100644
--- a/mllib-local/pom.xml
+++ b/mllib/common/pom.xml
@@ -22,26 +22,38 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.5.0-SNAPSHOT</version>
- <relativePath>../pom.xml</relativePath>
+ <relativePath>../../pom.xml</relativePath>
</parent>
- <artifactId>spark-mllib-local_2.12</artifactId>
+ <artifactId>spark-mllib-common_2.12</artifactId>
<properties>
- <sbt.project.name>mllib-local</sbt.project.name>
+ <sbt.project.name>mllib-common</sbt.project.name>
</properties>
<packaging>jar</packaging>
- <name>Spark Project ML Local Library</name>
+ <name>Spark Project ML Common</name>
<url>https://spark.apache.org/</url>
<dependencies>
<dependency>
- <groupId>org.scalanlp</groupId>
- <artifactId>breeze_${scala.binary.version}</artifactId>
+ <groupId>org.json4s</groupId>
+ <artifactId>json4s-jackson_${scala.binary.version}</artifactId>
</dependency>
<dependency>
- <groupId>org.apache.commons</groupId>
- <artifactId>commons-math3</artifactId>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-mllib-local_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <!--TODO: update the dependency once the catalyst refactoring is done-->
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
</dependency>
+
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
@@ -59,20 +71,18 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
</dependency>
-
- <!--
- This spark-tags test-dep is needed even though it isn't used in this module, otherwise testing-cmds that exclude
- them will yield errors.
- -->
<dependency>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ <artifactId>spark-mllib-local_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
-
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
@@ -81,9 +91,15 @@
<scope>test</scope>
</dependency>
+ <!--
+ This spark-tags test-dep is needed even though it isn't used in this module, otherwise testing-cmds that exclude
+ them will yield errors.
+ -->
<dependency>
- <groupId>dev.ludovic.netlib</groupId>
- <artifactId>blas</artifactId>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
</dependency>
</dependencies>
<build>
diff --git a/mllib/common/src/main/scala/org/apache/spark/ml/Predictor.scala b/mllib/common/src/main/scala/org/apache/spark/ml/Predictor.scala
new file mode 100644
index 00000000000..4fad2db81d7
--- /dev/null
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/Predictor.scala
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml
+
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.util.SchemaUtils
+import org.apache.spark.sql.types.{DataType, DoubleType, StructType}
+
+/**
+ * (private[ml]) Trait for parameters for prediction (regression and classification).
+ */
+private[ml] trait PredictorParams
+ extends Params
+ with HasLabelCol
+ with HasFeaturesCol
+ with HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema with the provided param map.
+ *
+ * @param schema
+ * input schema
+ * @param fitting
+ * whether this is in fitting
+ * @param featuresDataType
+ * SQL DataType for FeaturesType. E.g., `VectorUDT` for vector features.
+ * @return
+ * output schema
+ */
+ protected def validateAndTransformSchema(
+ schema: StructType,
+ fitting: Boolean,
+ featuresDataType: DataType): StructType = {
+ // TODO: Support casting Array[Double] and Array[Float] to Vector when FeaturesType = Vector
+ SchemaUtils.checkColumnType(schema, $(featuresCol), featuresDataType)
+ if (fitting) {
+ SchemaUtils.checkNumericType(schema, $(labelCol))
+
+ this match {
+ case p: HasWeightCol =>
+ if (isDefined(p.weightCol) && $(p.weightCol).nonEmpty) {
+ SchemaUtils.checkNumericType(schema, $(p.weightCol))
+ }
+ case _ =>
+ }
+ }
+ SchemaUtils.appendColumn(schema, $(predictionCol), DoubleType)
+ }
+}
+
diff --git a/mllib/src/main/scala/org/apache/spark/ml/attribute/AttributeGroup.scala b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/AttributeGroup.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/attribute/AttributeGroup.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/attribute/AttributeGroup.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/attribute/AttributeKeys.scala b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/AttributeKeys.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/attribute/AttributeKeys.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/attribute/AttributeKeys.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/attribute/AttributeType.scala b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/AttributeType.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/attribute/AttributeType.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/attribute/AttributeType.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/attribute/attributes.scala b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/attributes.scala
similarity index 99%
rename from mllib/src/main/scala/org/apache/spark/ml/attribute/attributes.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/attribute/attributes.scala
index f11cd865843..576c771d83b 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/attribute/attributes.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/attributes.scala
@@ -20,7 +20,6 @@ package org.apache.spark.ml.attribute
import scala.annotation.varargs
import org.apache.spark.sql.types.{DoubleType, Metadata, MetadataBuilder, NumericType, StructField}
-import org.apache.spark.util.collection.Utils
/**
* Abstract class for ML attributes.
@@ -339,7 +338,7 @@ class NominalAttribute private[ml] (
override def isNominal: Boolean = true
private lazy val valueToIndex: Map[String, Int] = {
- values.map(Utils.toMapWithIndex(_)).getOrElse(Map.empty)
+ values.map(_.zipWithIndex.toMap).getOrElse(Map.empty)
}
/** Index of a specific value. */
diff --git a/mllib/src/main/scala/org/apache/spark/ml/attribute/package-info.java b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/package-info.java
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/attribute/package-info.java
rename to mllib/common/src/main/scala/org/apache/spark/ml/attribute/package-info.java
diff --git a/mllib/src/main/scala/org/apache/spark/ml/attribute/package.scala b/mllib/common/src/main/scala/org/apache/spark/ml/attribute/package.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/attribute/package.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/attribute/package.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala b/mllib/common/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
similarity index 52%
copy from mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
copy to mllib/common/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
index 653ffb765f6..c81caddba73 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
@@ -15,34 +15,24 @@
* limitations under the License.
*/
-package org.apache.spark.ml.util
-
-import java.util.UUID
+package org.apache.spark.ml.classification
+import org.apache.spark.ml.PredictorParams
+import org.apache.spark.ml.linalg.VectorUDT
+import org.apache.spark.ml.param.shared.HasRawPredictionCol
+import org.apache.spark.ml.util.SchemaUtils
+import org.apache.spark.sql.types.{DataType, StructType}
/**
- * Trait for an object with an immutable unique ID that identifies itself and its derivatives.
- *
- * WARNING: There have not yet been final discussions on this API, so it may be broken in future
- * releases.
+ * (private[spark]) Params for classification.
*/
-trait Identifiable {
-
- /**
- * An immutable unique ID for the object and its derivatives.
- */
- val uid: String
-
- override def toString: String = uid
-}
-
-
-object Identifiable {
-
- /**
- * Returns a random UID that concatenates the given prefix, "_", and 12 random hex chars.
- */
- def randomUID(prefix: String): String = {
- prefix + "_" + UUID.randomUUID().toString.takeRight(12)
+private[spark] trait ClassifierParams extends PredictorParams with HasRawPredictionCol {
+
+ override protected def validateAndTransformSchema(
+ schema: StructType,
+ fitting: Boolean,
+ featuresDataType: DataType): StructType = {
+ val parentSchema = super.validateAndTransformSchema(schema, fitting, featuresDataType)
+ SchemaUtils.appendColumn(parentSchema, $(rawPredictionCol), new VectorUDT)
}
}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala b/mllib/common/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
similarity index 52%
copy from mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
copy to mllib/common/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
index 653ffb765f6..535c9e61fcd 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
@@ -15,34 +15,25 @@
* limitations under the License.
*/
-package org.apache.spark.ml.util
-
-import java.util.UUID
+package org.apache.spark.ml.classification
+import org.apache.spark.ml.linalg.VectorUDT
+import org.apache.spark.ml.param.shared._
+import org.apache.spark.ml.util.SchemaUtils
+import org.apache.spark.sql.types.{DataType, StructType}
/**
- * Trait for an object with an immutable unique ID that identifies itself and its derivatives.
- *
- * WARNING: There have not yet been final discussions on this API, so it may be broken in future
- * releases.
+ * (private[classification]) Params for probabilistic classification.
*/
-trait Identifiable {
-
- /**
- * An immutable unique ID for the object and its derivatives.
- */
- val uid: String
-
- override def toString: String = uid
-}
-
-
-object Identifiable {
-
- /**
- * Returns a random UID that concatenates the given prefix, "_", and 12 random hex chars.
- */
- def randomUID(prefix: String): String = {
- prefix + "_" + UUID.randomUUID().toString.takeRight(12)
+private[ml] trait ProbabilisticClassifierParams
+ extends ClassifierParams
+ with HasProbabilityCol
+ with HasThresholds {
+ override protected def validateAndTransformSchema(
+ schema: StructType,
+ fitting: Boolean,
+ featuresDataType: DataType): StructType = {
+ val parentSchema = super.validateAndTransformSchema(schema, fitting, featuresDataType)
+ SchemaUtils.appendColumn(parentSchema, $(probabilityCol), new VectorUDT)
}
}
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/package.scala b/mllib/common/src/main/scala/org/apache/spark/ml/feature/Instance.scala
similarity index 67%
copy from mllib/src/main/scala/org/apache/spark/mllib/api/python/package.scala
copy to mllib/common/src/main/scala/org/apache/spark/ml/feature/Instance.scala
index c67a6d3ae6c..7e754b151b7 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/api/python/package.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/feature/Instance.scala
@@ -15,11 +15,15 @@
* limitations under the License.
*/
-package org.apache.spark.mllib.api
+package org.apache.spark.ml.feature
+
+import org.apache.spark.ml.linalg._
/**
- * Internal support for MLlib Python API.
+ * Class that represents an instance of weighted data point with label and features.
*
- * @see [[org.apache.spark.mllib.api.python.PythonMLLibAPI]]
+ * @param label Label for this data point.
+ * @param weight The weight of this instance.
+ * @param features The vector of features for this data point.
*/
-package object python
+private[spark] case class Instance(label: Double, weight: Double, features: Vector)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala b/mllib/common/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/JsonMatrixConverter.scala b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/JsonMatrixConverter.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/linalg/JsonMatrixConverter.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/linalg/JsonMatrixConverter.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
similarity index 96%
rename from mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
index 27a67d561e3..197ec1a9936 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ml.linalg
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{GenericInternalRow, UnsafeArrayData}
-import org.apache.spark.sql.types._
+import org.apache.spark.sql.types.{ArrayType, BooleanType, ByteType, DoubleType, IntegerType, StructField, StructType, UserDefinedType}
/**
* User-defined type for [[Matrix]] in mllib-local which allows easy interaction with SQL
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/SQLDataTypes.scala b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/SQLDataTypes.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/linalg/SQLDataTypes.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/linalg/SQLDataTypes.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
similarity index 96%
rename from mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
index 302a94c8781..fbe22f3fc3a 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ml.linalg
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{GenericInternalRow, UnsafeArrayData}
-import org.apache.spark.sql.types._
+import org.apache.spark.sql.types.{ArrayType, ByteType, DoubleType, IntegerType, StructField, StructType, UserDefinedType}
/**
* User-defined type for [[Vector]] in mllib-local which allows easy interaction with SQL
diff --git a/mllib/src/main/scala/org/apache/spark/ml/param/params.scala b/mllib/common/src/main/scala/org/apache/spark/ml/param/params.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/param/params.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/param/params.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala b/mllib/common/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
similarity index 71%
copy from mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
copy to mllib/common/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
index 021d0b3e341..3a864fa394f 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
+++ b/mllib/common/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
@@ -17,16 +17,14 @@
package org.apache.spark.ml.param.shared
-import scala.concurrent.ExecutionContext
-
import org.apache.spark.ml.param.{IntParam, Params, ParamValidators}
-import org.apache.spark.util.ThreadUtils
/**
* Trait to define a level of parallelism for algorithms that are able to use
* multithreaded execution, and provide a thread-pool based execution context.
*/
private[ml] trait HasParallelism extends Params {
+ // TODO: Move HasParallelism into sharedParames
/**
* The number of threads to use when running parallel algorithms.
@@ -41,19 +39,4 @@ private[ml] trait HasParallelism extends Params {
/** @group expertGetParam */
def getParallelism: Int = $(parallelism)
-
- /**
- * Create a new execution context with a thread-pool that has a maximum number of threads
- * set to the value of [[parallelism]]. If this param is set to 1, a same-thread executor
- * will be used to run in serial.
- */
- private[ml] def getExecutionContext: ExecutionContext = {
- getParallelism match {
- case 1 =>
- ThreadUtils.sameThread
- case n =>
- ExecutionContext.fromExecutorService(ThreadUtils
- .newDaemonCachedThreadPool(s"${this.getClass.getSimpleName}-thread-pool", n))
- }
- }
}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala b/mllib/common/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala b/mllib/common/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/util/Identifiable.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/SchemaUtils.scala b/mllib/common/src/main/scala/org/apache/spark/ml/util/SchemaUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/SchemaUtils.scala
rename to mllib/common/src/main/scala/org/apache/spark/ml/util/SchemaUtils.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/attribute/AttributeGroupSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/attribute/AttributeGroupSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/attribute/AttributeGroupSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/attribute/AttributeGroupSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/attribute/AttributeSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/attribute/AttributeSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/attribute/AttributeSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/attribute/AttributeSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/linalg/JsonMatrixConverterSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/linalg/JsonMatrixConverterSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/linalg/JsonMatrixConverterSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/linalg/JsonMatrixConverterSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/linalg/JsonVectorConverterSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/linalg/JsonVectorConverterSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/linalg/JsonVectorConverterSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/linalg/JsonVectorConverterSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/linalg/MatrixUDTSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/linalg/MatrixUDTSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/linalg/MatrixUDTSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/linalg/MatrixUDTSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/linalg/SQLDataTypesSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/linalg/SQLDataTypesSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/linalg/SQLDataTypesSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/linalg/SQLDataTypesSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
similarity index 81%
copy from mllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
copy to mllib/common/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
index 67c64f762b2..582b38b3276 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
+++ b/mllib/common/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
@@ -18,10 +18,7 @@
package org.apache.spark.ml.linalg
import org.apache.spark.SparkFunSuite
-import org.apache.spark.ml.feature.LabeledPoint
-import org.apache.spark.sql.catalyst.JavaTypeInference
import org.apache.spark.sql.types._
-
class VectorUDTSuite extends SparkFunSuite {
test("preloaded VectorUDT") {
@@ -38,10 +35,4 @@ class VectorUDTSuite extends SparkFunSuite {
assert(udt.simpleString == "vector")
}
}
-
- test("JavaTypeInference with VectorUDT") {
- val (dataType, _) = JavaTypeInference.inferDataType(classOf[LabeledPoint])
- assert(dataType.asInstanceOf[StructType].fields.map(_.dataType)
- === Seq(new VectorUDT, DoubleType))
- }
}
diff --git a/mllib/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
similarity index 81%
rename from mllib/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
index 36e06091d24..cc94560969f 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
+++ b/mllib/common/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
@@ -17,13 +17,8 @@
package org.apache.spark.ml.param
-import java.io.{ByteArrayOutputStream, ObjectOutputStream}
-
import org.apache.spark.SparkFunSuite
-import org.apache.spark.ml.{Estimator, Transformer}
import org.apache.spark.ml.linalg.{Vector, Vectors}
-import org.apache.spark.ml.util.MyParams
-import org.apache.spark.sql.Dataset
class ParamsSuite extends SparkFunSuite {
@@ -379,77 +374,4 @@ class ParamsSuite extends SparkFunSuite {
val t3 = t.copy(ParamMap(t.maxIter -> 20))
assert(t3.isSet(t3.maxIter))
}
-
- test("Filtering ParamMap") {
- val params1 = new MyParams("my_params1")
- val params2 = new MyParams("my_params2")
- val paramMap = ParamMap(
- params1.intParam -> 1,
- params2.intParam -> 1,
- params1.doubleParam -> 0.2,
- params2.doubleParam -> 0.2)
- val filteredParamMap = paramMap.filter(params1)
-
- assert(filteredParamMap.size === 2)
- filteredParamMap.toSeq.foreach {
- case ParamPair(p, _) =>
- assert(p.parent === params1.uid)
- }
-
- // At the previous implementation of ParamMap#filter,
- // mutable.Map#filterKeys was used internally but
- // the return type of the method is not serializable (see SI-6654).
- // Now mutable.Map#filter is used instead of filterKeys and the return type is serializable.
- // So let's ensure serializability.
- val objOut = new ObjectOutputStream(new ByteArrayOutputStream())
- objOut.writeObject(filteredParamMap)
- }
-}
-
-object ParamsSuite extends SparkFunSuite {
-
- /**
- * Checks common requirements for `Params.params`:
- * - params are ordered by names
- * - param parent has the same UID as the object's UID
- * - param name is the same as the param method name
- * - obj.copy should return the same type as the obj
- */
- def checkParams(obj: Params): Unit = {
- val clazz = obj.getClass
-
- val params = obj.params
- val paramNames = params.map(_.name)
- require(paramNames === paramNames.sorted, "params must be ordered by names")
- params.foreach { p =>
- assert(p.parent === obj.uid)
- assert(obj.getParam(p.name) === p)
- // TODO: Check that setters return self, which needs special handling for generic types.
- }
-
- val copyMethod = clazz.getMethod("copy", classOf[ParamMap])
- val copyReturnType = copyMethod.getReturnType
- require(copyReturnType === obj.getClass,
- s"${clazz.getName}.copy should return ${clazz.getName} instead of ${copyReturnType.getName}.")
- }
-
- /**
- * Checks that the class throws an exception in case multiple exclusive params are set.
- * The params to be checked are passed as arguments with their value.
- */
- def testExclusiveParams(
- model: Params,
- dataset: Dataset[_],
- paramsAndValues: (String, Any)*): Unit = {
- val m = model.copy(ParamMap.empty)
- paramsAndValues.foreach { case (paramName, paramValue) =>
- m.set(m.getParam(paramName), paramValue)
- }
- intercept[IllegalArgumentException] {
- m match {
- case t: Transformer => t.transform(dataset)
- case e: Estimator[_] => e.fit(dataset)
- }
- }
- }
}
diff --git a/mllib/src/test/scala/org/apache/spark/ml/param/TestParams.scala b/mllib/common/src/test/scala/org/apache/spark/ml/param/TestParams.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/param/TestParams.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/param/TestParams.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/param/shared/SharedParamsSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/param/shared/SharedParamsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/param/shared/SharedParamsSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/param/shared/SharedParamsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/IdentifiableSuite.scala b/mllib/common/src/test/scala/org/apache/spark/ml/util/IdentifiableSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/IdentifiableSuite.scala
rename to mllib/common/src/test/scala/org/apache/spark/ml/util/IdentifiableSuite.scala
diff --git a/mllib/benchmarks/UDTSerializationBenchmark-jdk11-results.txt b/mllib/core/benchmarks/UDTSerializationBenchmark-jdk11-results.txt
similarity index 100%
rename from mllib/benchmarks/UDTSerializationBenchmark-jdk11-results.txt
rename to mllib/core/benchmarks/UDTSerializationBenchmark-jdk11-results.txt
diff --git a/mllib/benchmarks/UDTSerializationBenchmark-jdk17-results.txt b/mllib/core/benchmarks/UDTSerializationBenchmark-jdk17-results.txt
similarity index 100%
rename from mllib/benchmarks/UDTSerializationBenchmark-jdk17-results.txt
rename to mllib/core/benchmarks/UDTSerializationBenchmark-jdk17-results.txt
diff --git a/mllib/benchmarks/UDTSerializationBenchmark-results.txt b/mllib/core/benchmarks/UDTSerializationBenchmark-results.txt
similarity index 100%
rename from mllib/benchmarks/UDTSerializationBenchmark-results.txt
rename to mllib/core/benchmarks/UDTSerializationBenchmark-results.txt
diff --git a/mllib/pom.xml b/mllib/core/pom.xml
similarity index 93%
rename from mllib/pom.xml
rename to mllib/core/pom.xml
index 73af83c7586..a23fdc61e5e 100644
--- a/mllib/pom.xml
+++ b/mllib/core/pom.xml
@@ -22,7 +22,7 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.5.0-SNAPSHOT</version>
- <relativePath>../pom.xml</relativePath>
+ <relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>spark-mllib_2.12</artifactId>
@@ -84,6 +84,11 @@
<artifactId>spark-mllib-local_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-mllib-common_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib-local_${scala.binary.version}</artifactId>
@@ -91,6 +96,13 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-mllib-common_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
<!-- #if scala-2.13 --><!--
<dependency>
<groupId>org.scala-lang.modules</groupId>
diff --git a/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister b/mllib/core/src/main/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister
similarity index 100%
rename from mllib/src/main/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister
rename to mllib/core/src/main/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister
diff --git a/mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister b/mllib/core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
similarity index 100%
rename from mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
rename to mllib/core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/README b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/README
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/README
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/README
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/danish.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/danish.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/danish.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/danish.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/dutch.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/dutch.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/dutch.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/dutch.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/english.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/english.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/english.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/english.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/finnish.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/finnish.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/finnish.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/finnish.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/french.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/french.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/french.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/french.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/german.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/german.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/german.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/german.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/hungarian.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/hungarian.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/hungarian.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/hungarian.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/italian.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/italian.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/italian.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/italian.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/norwegian.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/norwegian.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/norwegian.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/norwegian.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/portuguese.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/portuguese.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/portuguese.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/portuguese.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/russian.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/russian.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/russian.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/russian.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/spanish.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/spanish.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/spanish.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/spanish.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/swedish.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/swedish.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/swedish.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/swedish.txt
diff --git a/mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/turkish.txt b/mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/turkish.txt
similarity index 100%
rename from mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/turkish.txt
rename to mllib/core/src/main/resources/org/apache/spark/ml/feature/stopwords/turkish.txt
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Estimator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/Estimator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/Estimator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Model.scala b/mllib/core/src/main/scala/org/apache/spark/ml/Model.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/Model.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/Model.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala b/mllib/core/src/main/scala/org/apache/spark/ml/Pipeline.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/Pipeline.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Predictor.scala b/mllib/core/src/main/scala/org/apache/spark/ml/Predictor.scala
similarity index 85%
rename from mllib/src/main/scala/org/apache/spark/ml/Predictor.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/Predictor.scala
index 9c6eb880c80..0153bf7b51b 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/Predictor.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/Predictor.scala
@@ -26,42 +26,6 @@ import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.{DataType, DoubleType, StructType}
-/**
- * (private[ml]) Trait for parameters for prediction (regression and classification).
- */
-private[ml] trait PredictorParams extends Params
- with HasLabelCol with HasFeaturesCol with HasPredictionCol {
-
- /**
- * Validates and transforms the input schema with the provided param map.
- *
- * @param schema input schema
- * @param fitting whether this is in fitting
- * @param featuresDataType SQL DataType for FeaturesType.
- * E.g., `VectorUDT` for vector features.
- * @return output schema
- */
- protected def validateAndTransformSchema(
- schema: StructType,
- fitting: Boolean,
- featuresDataType: DataType): StructType = {
- // TODO: Support casting Array[Double] and Array[Float] to Vector when FeaturesType = Vector
- SchemaUtils.checkColumnType(schema, $(featuresCol), featuresDataType)
- if (fitting) {
- SchemaUtils.checkNumericType(schema, $(labelCol))
-
- this match {
- case p: HasWeightCol =>
- if (isDefined(p.weightCol) && $(p.weightCol).nonEmpty) {
- SchemaUtils.checkNumericType(schema, $(p.weightCol))
- }
- case _ =>
- }
- }
- SchemaUtils.appendColumn(schema, $(predictionCol), DoubleType)
- }
-}
-
/**
* Abstraction for prediction problems (regression and classification). It accepts all NumericType
* labels and will automatically cast it to DoubleType in `fit()`. If this predictor supports
diff --git a/mllib/src/main/scala/org/apache/spark/ml/Transformer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/Transformer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/Transformer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/ann/BreezeUtil.scala b/mllib/core/src/main/scala/org/apache/spark/ml/ann/BreezeUtil.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/ann/BreezeUtil.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/ann/BreezeUtil.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/ann/Layer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/ann/Layer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/ann/LossFunction.scala b/mllib/core/src/main/scala/org/apache/spark/ml/ann/LossFunction.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/ann/LossFunction.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/ann/LossFunction.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/ClassificationSummary.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/ClassificationSummary.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/ClassificationSummary.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/ClassificationSummary.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
similarity index 91%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
index c46be175cb2..838149c2819 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/classification/Classifier.scala
@@ -18,29 +18,13 @@
package org.apache.spark.ml.classification
import org.apache.spark.annotation.Since
-import org.apache.spark.ml.{PredictionModel, Predictor, PredictorParams}
-import org.apache.spark.ml.linalg.{Vector, VectorUDT}
+import org.apache.spark.ml.{PredictionModel, Predictor}
+import org.apache.spark.ml.linalg.Vector
import org.apache.spark.ml.param.ParamMap
-import org.apache.spark.ml.param.shared.HasRawPredictionCol
import org.apache.spark.ml.util._
import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.sql.functions._
-import org.apache.spark.sql.types.{DataType, StructType}
-
-/**
- * (private[spark]) Params for classification.
- */
-private[spark] trait ClassifierParams
- extends PredictorParams with HasRawPredictionCol {
-
- override protected def validateAndTransformSchema(
- schema: StructType,
- fitting: Boolean,
- featuresDataType: DataType): StructType = {
- val parentSchema = super.validateAndTransformSchema(schema, fitting, featuresDataType)
- SchemaUtils.appendColumn(parentSchema, $(rawPredictionCol), new VectorUDT)
- }
-}
+import org.apache.spark.sql.types.StructType
/**
* Single-label binary or multiclass classification.
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
similarity index 99%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
index cf94c9fd36a..5bd27b3e809 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala
@@ -34,7 +34,7 @@ import org.apache.spark.ml._
import org.apache.spark.ml.attribute._
import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.ml.param.{Param, ParamMap, ParamPair, Params}
-import org.apache.spark.ml.param.shared.{HasParallelism, HasWeightCol}
+import org.apache.spark.ml.param.shared.{HasExecutionContext, HasWeightCol}
import org.apache.spark.ml.util._
import org.apache.spark.ml.util.Instrumentation.instrumented
import org.apache.spark.sql.{Column, DataFrame, Dataset, Row}
@@ -320,7 +320,8 @@ object OneVsRestModel extends MLReadable[OneVsRestModel] {
@Since("1.4.0")
final class OneVsRest @Since("1.4.0") (
@Since("1.4.0") override val uid: String)
- extends Estimator[OneVsRestModel] with OneVsRestParams with HasParallelism with MLWritable {
+ extends Estimator[OneVsRestModel] with OneVsRestParams
+ with HasExecutionContext with MLWritable {
@Since("1.4.0")
def this() = this(Identifiable.randomUID("oneVsRest"))
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
similarity index 93%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
index 1caaeccd7b0..dbbad54fc3a 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/classification/ProbabilisticClassifier.scala
@@ -18,28 +18,12 @@
package org.apache.spark.ml.classification
import org.apache.spark.annotation.Since
-import org.apache.spark.ml.linalg.{DenseVector, Vector, VectorUDT}
+import org.apache.spark.ml.linalg.{DenseVector, Vector}
import org.apache.spark.ml.param.ParamMap
-import org.apache.spark.ml.param.shared._
import org.apache.spark.ml.util.SchemaUtils
import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.sql.functions._
-import org.apache.spark.sql.types.{DataType, StructType}
-
-/**
- * (private[classification]) Params for probabilistic classification.
- */
-private[ml] trait ProbabilisticClassifierParams
- extends ClassifierParams with HasProbabilityCol with HasThresholds {
- override protected def validateAndTransformSchema(
- schema: StructType,
- fitting: Boolean,
- featuresDataType: DataType): StructType = {
- val parentSchema = super.validateAndTransformSchema(schema, fitting, featuresDataType)
- SchemaUtils.appendColumn(parentSchema, $(probabilityCol), new VectorUDT)
- }
-}
-
+import org.apache.spark.sql.types.StructType
/**
* Single-label binary or multiclass classifier which can output class conditional probabilities.
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala b/mllib/core/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala b/mllib/core/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/ClusteringSummary.scala b/mllib/core/src/main/scala/org/apache/spark/ml/clustering/ClusteringSummary.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/clustering/ClusteringSummary.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/clustering/ClusteringSummary.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala b/mllib/core/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala b/mllib/core/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala b/mllib/core/src/main/scala/org/apache/spark/ml/clustering/LDA.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/clustering/LDA.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala b/mllib/core/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/Evaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/Evaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/Evaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/Evaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/RankingEvaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/RankingEvaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/RankingEvaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/RankingEvaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/events.scala b/mllib/core/src/main/scala/org/apache/spark/ml/events.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/events.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/events.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSH.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSH.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSH.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSH.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Bucketizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Bucketizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Bucketizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Bucketizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/DCT.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/DCT.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/ElementwiseProduct.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/ElementwiseProduct.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/ElementwiseProduct.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/ElementwiseProduct.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/HashingTF.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/HashingTF.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/HashingTF.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/HashingTF.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/IDF.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/IDF.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Imputer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Imputer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Instance.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Instance.scala
similarity index 95%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Instance.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Instance.scala
index c237366ec5c..e1178ed24b1 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/Instance.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Instance.scala
@@ -22,16 +22,6 @@ import scala.collection.mutable
import org.apache.spark.ml.linalg._
import org.apache.spark.rdd.RDD
-/**
- * Class that represents an instance of weighted data point with label and features.
- *
- * @param label Label for this data point.
- * @param weight The weight of this instance.
- * @param features The vector of features for this data point.
- */
-private[spark] case class Instance(label: Double, weight: Double, features: Vector)
-
-
/**
* Class that represents an block of instance.
* If all weights are 1, then an empty array is stored.
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Interaction.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Interaction.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Interaction.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Interaction.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/LSH.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/LSH.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/NGram.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/NGram.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/PCA.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/PCA.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/RobustScaler.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/RobustScaler.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/RobustScaler.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/RobustScaler.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Selector.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Selector.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/VectorSizeHint.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorSizeHint.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/VectorSizeHint.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorSizeHint.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/VectorSlicer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorSlicer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/VectorSlicer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/VectorSlicer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java b/mllib/core/src/main/scala/org/apache/spark/ml/feature/package-info.java
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/package-info.java
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/package.scala b/mllib/core/src/main/scala/org/apache/spark/ml/feature/package.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/feature/package.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/feature/package.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala b/mllib/core/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/fpm/PrefixSpan.scala b/mllib/core/src/main/scala/org/apache/spark/ml/fpm/PrefixSpan.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/fpm/PrefixSpan.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/fpm/PrefixSpan.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/functions.scala b/mllib/core/src/main/scala/org/apache/spark/ml/functions.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/functions.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/functions.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/image/HadoopUtils.scala b/mllib/core/src/main/scala/org/apache/spark/ml/image/HadoopUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/image/HadoopUtils.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/image/HadoopUtils.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala b/mllib/core/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/NormalEquationSolver.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/NormalEquationSolver.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/NormalEquationSolver.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/NormalEquationSolver.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularization.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularization.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularization.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularization.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/loss/RDDLossFunction.scala b/mllib/core/src/main/scala/org/apache/spark/ml/optim/loss/RDDLossFunction.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/optim/loss/RDDLossFunction.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/optim/loss/RDDLossFunction.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/package-info.java b/mllib/core/src/main/scala/org/apache/spark/ml/package-info.java
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/package-info.java
rename to mllib/core/src/main/scala/org/apache/spark/ml/package-info.java
diff --git a/mllib/src/main/scala/org/apache/spark/ml/package.scala b/mllib/core/src/main/scala/org/apache/spark/ml/package.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/package.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/package.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala b/mllib/core/src/main/scala/org/apache/spark/ml/param/shared/HasExecutionContext.scala
similarity index 68%
rename from mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/param/shared/HasExecutionContext.scala
index 021d0b3e341..fe61416202b 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/param/shared/HasParallelism.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/param/shared/HasExecutionContext.scala
@@ -19,28 +19,9 @@ package org.apache.spark.ml.param.shared
import scala.concurrent.ExecutionContext
-import org.apache.spark.ml.param.{IntParam, Params, ParamValidators}
import org.apache.spark.util.ThreadUtils
-/**
- * Trait to define a level of parallelism for algorithms that are able to use
- * multithreaded execution, and provide a thread-pool based execution context.
- */
-private[ml] trait HasParallelism extends Params {
-
- /**
- * The number of threads to use when running parallel algorithms.
- * Default is 1 for serial execution
- *
- * @group expertParam
- */
- val parallelism = new IntParam(this, "parallelism",
- "the number of threads to use when running parallel algorithms", ParamValidators.gtEq(1))
-
- setDefault(parallelism -> 1)
-
- /** @group expertGetParam */
- def getParallelism: Int = $(parallelism)
+private[ml] trait HasExecutionContext extends HasParallelism {
/**
* Create a new execution context with a thread-pool that has a maximum number of threads
diff --git a/mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala b/mllib/core/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala b/mllib/core/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/ALSWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/ALSWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/ALSWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/ALSWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/FPGrowthWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/FPGrowthWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/FPGrowthWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/FPGrowthWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/GeneralizedLinearRegressionWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/GeneralizedLinearRegressionWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/GeneralizedLinearRegressionWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/GeneralizedLinearRegressionWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/KSTestWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/KSTestWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/KSTestWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/KSTestWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/MultilayerPerceptronClassifierWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/MultilayerPerceptronClassifierWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/MultilayerPerceptronClassifierWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/MultilayerPerceptronClassifierWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/NaiveBayesWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/NaiveBayesWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/NaiveBayesWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/NaiveBayesWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/PowerIterationClusteringWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/PowerIterationClusteringWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/PowerIterationClusteringWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/PowerIterationClusteringWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/RWrappers.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/RWrappers.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/RWrappers.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/RWrappers.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala b/mllib/core/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/Regressor.scala b/mllib/core/src/main/scala/org/apache/spark/ml/regression/Regressor.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/regression/Regressor.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/regression/Regressor.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala b/mllib/core/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala b/mllib/core/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala b/mllib/core/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala b/mllib/core/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala b/mllib/core/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala b/mllib/core/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/ANOVATest.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/ANOVATest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/ANOVATest.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/ANOVATest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/Correlation.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/Correlation.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/Correlation.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/Correlation.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/FValueTest.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/FValueTest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/FValueTest.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/FValueTest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTest.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTest.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/MultiClassSummarizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/MultiClassSummarizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/MultiClassSummarizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/MultiClassSummarizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala b/mllib/core/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/Node.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/Node.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/Node.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/Node.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/Split.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/Split.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/Split.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/Split.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/BaggedPoint.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/BaggedPoint.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/BaggedPoint.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/BaggedPoint.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/DTStatsAggregator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/DTStatsAggregator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/DTStatsAggregator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/DTStatsAggregator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/TimeTracker.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/TimeTracker.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/TimeTracker.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/TimeTracker.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/TreePoint.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/TreePoint.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/impl/TreePoint.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/impl/TreePoint.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tree/treeParams.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tree/treeParams.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
similarity index 99%
rename from mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
index e04a8c1389b..1cc7faef19a 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
@@ -31,7 +31,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.ml.{Estimator, Model}
import org.apache.spark.ml.evaluation.Evaluator
import org.apache.spark.ml.param.{IntParam, Param, ParamMap, ParamValidators}
-import org.apache.spark.ml.param.shared.{HasCollectSubModels, HasParallelism}
+import org.apache.spark.ml.param.shared.{HasCollectSubModels, HasExecutionContext}
import org.apache.spark.ml.util._
import org.apache.spark.ml.util.Instrumentation.instrumented
import org.apache.spark.mllib.util.MLUtils
@@ -79,7 +79,7 @@ private[ml] trait CrossValidatorParams extends ValidatorParams {
@Since("1.2.0")
class CrossValidator @Since("1.2.0") (@Since("1.4.0") override val uid: String)
extends Estimator[CrossValidatorModel]
- with CrossValidatorParams with HasParallelism with HasCollectSubModels
+ with CrossValidatorParams with HasExecutionContext with HasCollectSubModels
with MLWritable with Logging {
@Since("1.2.0")
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tuning/ParamGridBuilder.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tuning/ParamGridBuilder.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tuning/ParamGridBuilder.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tuning/ParamGridBuilder.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
similarity index 99%
rename from mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
index 4a6d5164aa0..5275ae41231 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
+++ b/mllib/core/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
@@ -32,7 +32,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.ml.{Estimator, Model}
import org.apache.spark.ml.evaluation.Evaluator
import org.apache.spark.ml.param.{DoubleParam, ParamMap, ParamValidators}
-import org.apache.spark.ml.param.shared.{HasCollectSubModels, HasParallelism}
+import org.apache.spark.ml.param.shared.{HasCollectSubModels, HasExecutionContext}
import org.apache.spark.ml.util._
import org.apache.spark.ml.util.Instrumentation.instrumented
import org.apache.spark.sql.{DataFrame, Dataset}
@@ -67,7 +67,7 @@ private[ml] trait TrainValidationSplitParams extends ValidatorParams {
@Since("1.5.0")
class TrainValidationSplit @Since("1.5.0") (@Since("1.5.0") override val uid: String)
extends Estimator[TrainValidationSplitModel]
- with TrainValidationSplitParams with HasParallelism with HasCollectSubModels
+ with TrainValidationSplitParams with HasExecutionContext with HasCollectSubModels
with MLWritable with Logging {
@Since("1.5.0")
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala b/mllib/core/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala b/mllib/core/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/HasTrainingSummary.scala b/mllib/core/src/main/scala/org/apache/spark/ml/util/HasTrainingSummary.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/HasTrainingSummary.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/util/HasTrainingSummary.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala b/mllib/core/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/MetadataUtils.scala b/mllib/core/src/main/scala/org/apache/spark/ml/util/MetadataUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/MetadataUtils.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/util/MetadataUtils.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala b/mllib/core/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/stopwatches.scala b/mllib/core/src/main/scala/org/apache/spark/ml/util/stopwatches.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/ml/util/stopwatches.scala
rename to mllib/core/src/main/scala/org/apache/spark/ml/util/stopwatches.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/JavaPackage.java b/mllib/core/src/main/scala/org/apache/spark/mllib/JavaPackage.java
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/JavaPackage.java
rename to mllib/core/src/main/scala/org/apache/spark/mllib/JavaPackage.java
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/FPGrowthModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/FPGrowthModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/FPGrowthModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/FPGrowthModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/GaussianMixtureModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/GaussianMixtureModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/GaussianMixtureModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/GaussianMixtureModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/LDAModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/LDAModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/LDAModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/LDAModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/MatrixFactorizationModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/MatrixFactorizationModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/MatrixFactorizationModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/MatrixFactorizationModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PowerIterationClusteringModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/PowerIterationClusteringModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/PowerIterationClusteringModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/PowerIterationClusteringModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PrefixSpanModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/PrefixSpanModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/PrefixSpanModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/PrefixSpanModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/Word2VecModelWrapper.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/api/python/package.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/api/python/package.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/api/python/package.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/api/python/package.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/ClassificationModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/classification/ClassificationModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/classification/ClassificationModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/classification/ClassificationModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/classification/SVM.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionWithSGD.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionWithSGD.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionWithSGD.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionWithSGD.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/classification/impl/GLMClassificationModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/classification/impl/GLMClassificationModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/classification/impl/GLMClassificationModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/classification/impl/GLMClassificationModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/DistanceMeasure.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/DistanceMeasure.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/DistanceMeasure.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/DistanceMeasure.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixture.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixture.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixture.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixture.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixtureModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixtureModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixtureModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixtureModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAUtils.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDAUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAUtils.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LDAUtils.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/AreaUnderCurve.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/AreaUnderCurve.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/AreaUnderCurve.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/AreaUnderCurve.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RegressionMetrics.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/RegressionMetrics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/RegressionMetrics.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/RegressionMetrics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryClassificationMetricComputers.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryClassificationMetricComputers.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryClassificationMetricComputers.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryClassificationMetricComputers.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryConfusionMatrix.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryConfusionMatrix.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryConfusionMatrix.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryConfusionMatrix.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryLabelCounter.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryLabelCounter.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryLabelCounter.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/evaluation/binary/BinaryLabelCounter.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/ElementwiseProduct.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/ElementwiseProduct.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/ElementwiseProduct.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/ElementwiseProduct.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/IDF.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/IDF.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Normalizer.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/Normalizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/Normalizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/Normalizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/PCA.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/PCA.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/PCA.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/PCA.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/StandardScaler.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/StandardScaler.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/StandardScaler.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/StandardScaler.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/VectorTransformer.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/VectorTransformer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/VectorTransformer.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/VectorTransformer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/fpm/AssociationRules.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/fpm/AssociationRules.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/fpm/AssociationRules.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/fpm/AssociationRules.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPTree.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/fpm/FPTree.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/fpm/FPTree.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/fpm/FPTree.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/fpm/LocalPrefixSpan.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/fpm/LocalPrefixSpan.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/fpm/LocalPrefixSpan.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/fpm/LocalPrefixSpan.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/ARPACK.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/ARPACK.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/ARPACK.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/ARPACK.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/CholeskyDecomposition.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/CholeskyDecomposition.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/CholeskyDecomposition.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/CholeskyDecomposition.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/LAPACK.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/LAPACK.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/LAPACK.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/LAPACK.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/SingularValueDecomposition.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/SingularValueDecomposition.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/SingularValueDecomposition.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/SingularValueDecomposition.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrix.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrix.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrix.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrix.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/DistributedMatrix.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/DistributedMatrix.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/DistributedMatrix.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/DistributedMatrix.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrix.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrix.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrix.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrix.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/Gradient.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/optimization/Gradient.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/optimization/Gradient.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/optimization/Gradient.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/NNLS.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/optimization/NNLS.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/optimization/NNLS.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/optimization/NNLS.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/Optimizer.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/optimization/Optimizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/optimization/Optimizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/optimization/Optimizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/Updater.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/optimization/Updater.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/optimization/Updater.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/optimization/Updater.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/package-info.java b/mllib/core/src/main/scala/org/apache/spark/mllib/package-info.java
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/package-info.java
rename to mllib/core/src/main/scala/org/apache/spark/mllib/package-info.java
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/package.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/package.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/package.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/package.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/pmml/PMMLExportable.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/pmml/PMMLExportable.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/pmml/PMMLExportable.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/pmml/PMMLExportable.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExport.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExport.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExport.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExport.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExport.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExport.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExport.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExport.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExport.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExport.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExport.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExport.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactory.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactory.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactory.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactory.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/random/RandomDataGenerator.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/random/RandomDataGenerator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/random/RandomDataGenerator.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/random/RandomDataGenerator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/random/RandomRDDs.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/random/RandomRDDs.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/random/RandomRDDs.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/random/RandomRDDs.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctions.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctions.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctions.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctions.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/rdd/RandomRDD.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/rdd/RandomRDD.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/rdd/RandomRDD.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/rdd/RandomRDD.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/rdd/SlidingRDD.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/rdd/SlidingRDD.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/rdd/SlidingRDD.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/rdd/SlidingRDD.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/LabeledPoint.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/LabeledPoint.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/LabeledPoint.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/LabeledPoint.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/RegressionModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/RegressionModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/RegressionModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/RegressionModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/RidgeRegression.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/RidgeRegression.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/RidgeRegression.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/RidgeRegression.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionWithSGD.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionWithSGD.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionWithSGD.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionWithSGD.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/KernelDensity.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/KernelDensity.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/KernelDensity.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/KernelDensity.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateStatisticalSummary.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/MultivariateStatisticalSummary.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateStatisticalSummary.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/MultivariateStatisticalSummary.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/Correlation.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/correlation/Correlation.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/Correlation.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/correlation/Correlation.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/PearsonCorrelation.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/correlation/PearsonCorrelation.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/PearsonCorrelation.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/correlation/PearsonCorrelation.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmanCorrelation.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmanCorrelation.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmanCorrelation.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmanCorrelation.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussian.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussian.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussian.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussian.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/test/KolmogorovSmirnovTest.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/KolmogorovSmirnovTest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/test/KolmogorovSmirnovTest.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/KolmogorovSmirnovTest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/StreamingTestMethod.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/RandomForest.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/RandomForest.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/RandomForest.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/RandomForest.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Algo.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/Algo.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Algo.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/Algo.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/BoostingStrategy.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/EnsembleCombiningStrategy.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/EnsembleCombiningStrategy.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/EnsembleCombiningStrategy.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/EnsembleCombiningStrategy.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/FeatureType.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/FeatureType.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/FeatureType.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/FeatureType.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/QuantileStrategy.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/QuantileStrategy.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/QuantileStrategy.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/QuantileStrategy.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurities.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurities.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurities.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurities.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurity.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurity.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurity.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Impurity.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Variance.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Variance.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Variance.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/impurity/Variance.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/AbsoluteError.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/loss/LogLoss.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/LogLoss.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/loss/LogLoss.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/LogLoss.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/loss/Loss.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/Loss.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/loss/Loss.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/Loss.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/loss/Losses.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/Losses.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/loss/Losses.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/Losses.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/loss/SquaredError.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/SquaredError.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/loss/SquaredError.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/loss/SquaredError.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/InformationGainStats.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/InformationGainStats.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/model/InformationGainStats.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/InformationGainStats.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/Node.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/Node.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/model/Node.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/Node.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/Predict.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/Predict.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/model/Predict.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/Predict.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/Split.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/Split.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/model/Split.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/Split.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/package.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/tree/package.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/tree/package.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/tree/package.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/KMeansDataGenerator.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/KMeansDataGenerator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/KMeansDataGenerator.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/KMeansDataGenerator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/LogisticRegressionDataGenerator.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/LogisticRegressionDataGenerator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/LogisticRegressionDataGenerator.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/LogisticRegressionDataGenerator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/MFDataGenerator.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/MFDataGenerator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/MFDataGenerator.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/MFDataGenerator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/NumericParser.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/NumericParser.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/NumericParser.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/NumericParser.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/modelSaveLoad.scala b/mllib/core/src/main/scala/org/apache/spark/mllib/util/modelSaveLoad.scala
similarity index 100%
rename from mllib/src/main/scala/org/apache/spark/mllib/util/modelSaveLoad.scala
rename to mllib/core/src/main/scala/org/apache/spark/mllib/util/modelSaveLoad.scala
diff --git a/mllib/src/test/java/org/apache/spark/SharedSparkSession.java b/mllib/core/src/test/java/org/apache/spark/SharedSparkSession.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/SharedSparkSession.java
rename to mllib/core/src/test/java/org/apache/spark/SharedSparkSession.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/JavaPipelineSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/JavaPipelineSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/JavaPipelineSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/JavaPipelineSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/attribute/JavaAttributeGroupSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/attribute/JavaAttributeGroupSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/attribute/JavaAttributeGroupSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/attribute/JavaAttributeGroupSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/attribute/JavaAttributeSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/attribute/JavaAttributeSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/attribute/JavaAttributeSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/attribute/JavaAttributeSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaDecisionTreeClassifierSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaDecisionTreeClassifierSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaDecisionTreeClassifierSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaDecisionTreeClassifierSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaGBTClassifierSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaGBTClassifierSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaGBTClassifierSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaGBTClassifierSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaLogisticRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaLogisticRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaLogisticRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaLogisticRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaMultilayerPerceptronClassifierSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaMultilayerPerceptronClassifierSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaMultilayerPerceptronClassifierSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaMultilayerPerceptronClassifierSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaNaiveBayesSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaNaiveBayesSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaNaiveBayesSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaNaiveBayesSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaOneVsRestSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaOneVsRestSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaOneVsRestSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaOneVsRestSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/clustering/JavaKMeansSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/clustering/JavaKMeansSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/clustering/JavaKMeansSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/clustering/JavaKMeansSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaBucketizerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaBucketizerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaBucketizerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaBucketizerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaDCTSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaDCTSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaDCTSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaDCTSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaHashingTFSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaHashingTFSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaHashingTFSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaHashingTFSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaNormalizerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaNormalizerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaNormalizerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaNormalizerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaPCASuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaPCASuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaPCASuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaPCASuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaPolynomialExpansionSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaPolynomialExpansionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaPolynomialExpansionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaPolynomialExpansionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaStandardScalerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaStandardScalerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaStandardScalerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaStandardScalerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaStopWordsRemoverSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaStopWordsRemoverSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaStopWordsRemoverSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaStopWordsRemoverSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaStringIndexerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaStringIndexerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaStringIndexerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaStringIndexerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaTokenizerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaTokenizerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaTokenizerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaTokenizerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaVectorAssemblerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaVectorAssemblerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaVectorAssemblerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaVectorAssemblerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaVectorIndexerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaVectorIndexerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaVectorIndexerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaVectorIndexerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaVectorSlicerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaVectorSlicerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaVectorSlicerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaVectorSlicerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/feature/JavaWord2VecSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/feature/JavaWord2VecSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/feature/JavaWord2VecSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/feature/JavaWord2VecSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/linalg/JavaSQLDataTypesSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/linalg/JavaSQLDataTypesSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/linalg/JavaSQLDataTypesSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/linalg/JavaSQLDataTypesSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/param/JavaParamsSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/param/JavaParamsSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/param/JavaParamsSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/param/JavaParamsSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/param/JavaTestParams.java b/mllib/core/src/test/java/org/apache/spark/ml/param/JavaTestParams.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/param/JavaTestParams.java
rename to mllib/core/src/test/java/org/apache/spark/ml/param/JavaTestParams.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/regression/JavaDecisionTreeRegressorSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/regression/JavaDecisionTreeRegressorSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/regression/JavaDecisionTreeRegressorSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/regression/JavaDecisionTreeRegressorSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/regression/JavaGBTRegressorSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/regression/JavaGBTRegressorSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/regression/JavaGBTRegressorSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/regression/JavaGBTRegressorSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/regression/JavaLinearRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/regression/JavaLinearRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/regression/JavaLinearRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/regression/JavaLinearRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/regression/JavaRandomForestRegressorSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/regression/JavaRandomForestRegressorSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/regression/JavaRandomForestRegressorSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/regression/JavaRandomForestRegressorSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/source/libsvm/JavaLibSVMRelationSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/source/libsvm/JavaLibSVMRelationSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/source/libsvm/JavaLibSVMRelationSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/source/libsvm/JavaLibSVMRelationSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/stat/JavaKolmogorovSmirnovTestSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/stat/JavaKolmogorovSmirnovTestSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/stat/JavaKolmogorovSmirnovTestSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/stat/JavaKolmogorovSmirnovTestSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/stat/JavaSummarizerSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/stat/JavaSummarizerSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/stat/JavaSummarizerSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/stat/JavaSummarizerSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/tuning/JavaCrossValidatorSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/tuning/JavaCrossValidatorSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/tuning/JavaCrossValidatorSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/tuning/JavaCrossValidatorSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java b/mllib/core/src/test/java/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java
rename to mllib/core/src/test/java/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/classification/JavaLogisticRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaLogisticRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/classification/JavaLogisticRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaLogisticRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/classification/JavaNaiveBayesSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaNaiveBayesSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/classification/JavaNaiveBayesSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaNaiveBayesSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/classification/JavaSVMSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaSVMSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/classification/JavaSVMSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaSVMSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/classification/JavaStreamingLogisticRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaStreamingLogisticRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/classification/JavaStreamingLogisticRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/classification/JavaStreamingLogisticRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/clustering/JavaBisectingKMeansSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaBisectingKMeansSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/clustering/JavaBisectingKMeansSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaBisectingKMeansSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/clustering/JavaGaussianMixtureSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaGaussianMixtureSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/clustering/JavaGaussianMixtureSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaGaussianMixtureSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/clustering/JavaKMeansSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaKMeansSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/clustering/JavaKMeansSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaKMeansSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/clustering/JavaLDASuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaLDASuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/clustering/JavaLDASuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaLDASuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/clustering/JavaStreamingKMeansSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaStreamingKMeansSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/clustering/JavaStreamingKMeansSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/clustering/JavaStreamingKMeansSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/evaluation/JavaRankingMetricsSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/evaluation/JavaRankingMetricsSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/evaluation/JavaRankingMetricsSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/evaluation/JavaRankingMetricsSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/feature/JavaTfIdfSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/feature/JavaTfIdfSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/feature/JavaTfIdfSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/feature/JavaTfIdfSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/feature/JavaWord2VecSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/feature/JavaWord2VecSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/feature/JavaWord2VecSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/feature/JavaWord2VecSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/fpm/JavaAssociationRulesSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/fpm/JavaAssociationRulesSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/fpm/JavaAssociationRulesSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/fpm/JavaAssociationRulesSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/fpm/JavaFPGrowthSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/fpm/JavaFPGrowthSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/fpm/JavaFPGrowthSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/fpm/JavaFPGrowthSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/fpm/JavaPrefixSpanSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/fpm/JavaPrefixSpanSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/fpm/JavaPrefixSpanSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/fpm/JavaPrefixSpanSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/linalg/JavaMatricesSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/linalg/JavaMatricesSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/linalg/JavaMatricesSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/linalg/JavaMatricesSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/linalg/JavaVectorsSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/linalg/JavaVectorsSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/linalg/JavaVectorsSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/linalg/JavaVectorsSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/random/JavaRandomRDDsSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/random/JavaRandomRDDsSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/random/JavaRandomRDDsSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/random/JavaRandomRDDsSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/recommendation/JavaALSSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/recommendation/JavaALSSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/recommendation/JavaALSSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/recommendation/JavaALSSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/regression/JavaIsotonicRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaIsotonicRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/regression/JavaIsotonicRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaIsotonicRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/regression/JavaLassoSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaLassoSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/regression/JavaLassoSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaLassoSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/regression/JavaLinearRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaLinearRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/regression/JavaLinearRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaLinearRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/regression/JavaRidgeRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaRidgeRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/regression/JavaRidgeRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaRidgeRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/regression/JavaStreamingLinearRegressionSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaStreamingLinearRegressionSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/regression/JavaStreamingLinearRegressionSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/regression/JavaStreamingLinearRegressionSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/stat/JavaStatisticsSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/stat/JavaStatisticsSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/stat/JavaStatisticsSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/stat/JavaStatisticsSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/tree/JavaDecisionTreeSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/tree/JavaDecisionTreeSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/tree/JavaDecisionTreeSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/tree/JavaDecisionTreeSuite.java
diff --git a/mllib/src/test/java/org/apache/spark/mllib/util/JavaMLUtilsSuite.java b/mllib/core/src/test/java/org/apache/spark/mllib/util/JavaMLUtilsSuite.java
similarity index 100%
rename from mllib/src/test/java/org/apache/spark/mllib/util/JavaMLUtilsSuite.java
rename to mllib/core/src/test/java/org/apache/spark/mllib/util/JavaMLUtilsSuite.java
diff --git a/mllib/src/test/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister b/mllib/core/src/test/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister
similarity index 100%
rename from mllib/src/test/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister
rename to mllib/core/src/test/resources/META-INF/services/org.apache.spark.ml.util.MLFormatRegister
diff --git a/mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg b/mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg
rename to mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg
diff --git a/mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-01/not-image.txt b/mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-01/not-image.txt
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-01/not-image.txt
rename to mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-01/not-image.txt
diff --git a/mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-02/54893.jpg b/mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-02/54893.jpg
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-02/54893.jpg
rename to mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-02/54893.jpg
diff --git a/mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg b/mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg
rename to mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg
diff --git a/mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg b/mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg
rename to mllib/core/src/test/resources/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg
diff --git a/mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA.png b/mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA.png
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA.png
rename to mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA.png
diff --git a/mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png b/mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png
rename to mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png
diff --git a/mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg b/mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg
rename to mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg
diff --git a/mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg b/mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg
similarity index 100%
rename from mllib/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg
rename to mllib/core/src/test/resources/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg
diff --git a/mllib/src/test/resources/iris_libsvm.txt b/mllib/core/src/test/resources/iris_libsvm.txt
similarity index 100%
rename from mllib/src/test/resources/iris_libsvm.txt
rename to mllib/core/src/test/resources/iris_libsvm.txt
diff --git a/mllib/src/test/resources/log4j2.properties b/mllib/core/src/test/resources/log4j2.properties
similarity index 100%
rename from mllib/src/test/resources/log4j2.properties
rename to mllib/core/src/test/resources/log4j2.properties
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/data/.part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/.part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/data/.part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/.part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/data/part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/data/part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/data/part-00000-bd7ae42f-c890-406c-894c-ca4eac67c690-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtc-2.4.7/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/dtc-2.4.7/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/data/.part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/.part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/data/.part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/.part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/data/part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/data/part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/data/part-00000-39b027f0-a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/dtr-2.4.7/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/dtr-2.4.7/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/data/.part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/.part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/data/.part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/.part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/data/part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/data/part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/data/part-00000-dacbde64-c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/.part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/.part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/.part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/.part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/gbtc-2.4.7/treesMetadata/part-00000-81137d9f-31e3-4a90-813c-ddc394101e21-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/data/.part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/.part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/data/.part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/.part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/data/part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/data/part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/data/part-00000-3b5433ff-d346-4511-9aab-639288bfae6d-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/.part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/.part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/.part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/.part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/gbtr-2.4.7/treesMetadata/part-00000-6b9124f5-87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/hashingTF-2.4.4/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/hashingTF-2.4.4/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/data/.part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/.part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/data/.part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/.part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/data/part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/data/part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/data/part-00000-fa18aaf6-d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/mlp-2.4.4/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/mlp-2.4.4/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/data/.part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/.part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/data/.part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/.part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/data/part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/data/part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/data/part-00000-e41a7b98-91f8-4485-b112-25b4b11c9009-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/.part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/.part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/.part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/.part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/rfc-2.4.7/treesMetadata/part-00000-21082d24-b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/data/.part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/.part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/data/.part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/.part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/data/part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/data/part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/data/part-00000-4a69607d-6edb-40fc-b681-981caaeca996-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/metadata/part-00000
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/.part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/.part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/.part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/.part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/rfr-2.4.7/treesMetadata/part-00000-dfe4db51-d349-447a-9b86-d95edaabcde8-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/.part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet.crc b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/.part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/.part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet.crc
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/.part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet.crc
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/_SUCCESS b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/data/part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/data/part-00000-f09b03f6-6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/._SUCCESS.crc b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/._SUCCESS.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/._SUCCESS.crc
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/._SUCCESS.crc
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/.part-00000.crc b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/.part-00000.crc
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/.part-00000.crc
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/.part-00000.crc
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/_SUCCESS b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/_SUCCESS
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/_SUCCESS
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/_SUCCESS
diff --git a/mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/part-00000 b/mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/part-00000
similarity index 100%
rename from mllib/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/part-00000
rename to mllib/core/src/test/resources/ml-models/strIndexerModel-2.4.4/metadata/part-00000
diff --git a/mllib/src/test/scala/org/apache/spark/ml/FunctionsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/FunctionsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/FunctionsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/FunctionsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/PipelineSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/PredictorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/PredictorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/PredictorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/PredictorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/ann/ANNSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/ann/GradientSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/ann/GradientSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/ann/GradientSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/ann/GradientSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/ClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/ClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/ClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/ClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/FMClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/FMClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/FMClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/FMClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/GBTClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/GBTClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/GBTClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/GBTClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/LinearSVCSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/LinearSVCSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/LinearSVCSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/LinearSVCSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala
similarity index 99%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala
index bed45fc68f4..8d964e09575 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala
+++ b/mllib/core/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala
@@ -72,10 +72,10 @@ class NaiveBayesSuite extends MLTest with DefaultReadWriteTest {
gaussianDataset = generateGaussianNaiveBayesInput(pi, theta2, sigma, 1000, seed).toDF()
gaussianDataset2 = spark.read.format("libsvm")
- .load("../data/mllib/sample_multiclass_classification_data.txt")
+ .load("../../data/mllib/sample_multiclass_classification_data.txt")
complementDataset = spark.read.format("libsvm")
- .load("../data/mllib/sample_libsvm_data.txt")
+ .load("../../data/mllib/sample_libsvm_data.txt")
}
def validatePrediction(predictionAndLabels: Seq[Row]): Unit = {
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/ProbabilisticClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/ProbabilisticClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/ProbabilisticClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/ProbabilisticClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/classification/RandomForestClassifierSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/classification/RandomForestClassifierSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/classification/RandomForestClassifierSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/classification/RandomForestClassifierSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/clustering/GaussianMixtureSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/clustering/KMeansSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/clustering/KMeansSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/clustering/KMeansSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/clustering/KMeansSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/clustering/LDASuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/clustering/LDASuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/clustering/LDASuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/clustering/LDASuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/clustering/PowerIterationClusteringSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/clustering/PowerIterationClusteringSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/clustering/PowerIterationClusteringSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/clustering/PowerIterationClusteringSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/evaluation/ClusteringEvaluatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/evaluation/ClusteringEvaluatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/evaluation/ClusteringEvaluatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/evaluation/ClusteringEvaluatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/evaluation/RankingEvaluatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/evaluation/RankingEvaluatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/evaluation/RankingEvaluatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/evaluation/RankingEvaluatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/evaluation/RegressionEvaluatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/evaluation/RegressionEvaluatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/evaluation/RegressionEvaluatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/evaluation/RegressionEvaluatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/BinarizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/BinarizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/BinarizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/BinarizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSHSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/BucketizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/BucketizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/BucketizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/BucketizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/CountVectorizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/CountVectorizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/CountVectorizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/CountVectorizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/DCTSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/DCTSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/DCTSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/DCTSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/ElementwiseProductSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/ElementwiseProductSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/ElementwiseProductSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/ElementwiseProductSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/HashingTFSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/HashingTFSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/HashingTFSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/HashingTFSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/IDFSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/ImputerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/ImputerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/ImputerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/ImputerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/InstanceSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/InstanceSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/InstanceSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/InstanceSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/InteractionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/InteractionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/InteractionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/InteractionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/LSHTest.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/LSHTest.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/LSHTest.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/LSHTest.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/LabeledPointSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/LabeledPointSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/LabeledPointSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/LabeledPointSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/MaxAbsScalerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/MaxAbsScalerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/MaxAbsScalerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/MaxAbsScalerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/MinHashLSHSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/MinHashLSHSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/MinHashLSHSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/MinHashLSHSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/NormalizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/NormalizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/NormalizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/NormalizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/OneHotEncoderSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/OneHotEncoderSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/OneHotEncoderSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/OneHotEncoderSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/PCASuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/PCASuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/PCASuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/PCASuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/PolynomialExpansionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/PolynomialExpansionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/PolynomialExpansionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/PolynomialExpansionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaParserSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/RFormulaParserSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaParserSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/RFormulaParserSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/RobustScalerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/RobustScalerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/RobustScalerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/RobustScalerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/StandardScalerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/StandardScalerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/StandardScalerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/StandardScalerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/StopWordsRemoverSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/StopWordsRemoverSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/StopWordsRemoverSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/StopWordsRemoverSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/TokenizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/UnivariateFeatureSelectorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/UnivariateFeatureSelectorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/UnivariateFeatureSelectorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/UnivariateFeatureSelectorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/VarianceThresholdSelectorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/VarianceThresholdSelectorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/VarianceThresholdSelectorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/VarianceThresholdSelectorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorAssemblerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorAssemblerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/VectorAssemblerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorAssemblerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorSizeHintSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorSizeHintSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/VectorSizeHintSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorSizeHintSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorSlicerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorSlicerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/VectorSlicerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/VectorSlicerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/Word2VecSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/feature/Word2VecSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/feature/Word2VecSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/feature/Word2VecSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/fpm/PrefixSpanSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/fpm/PrefixSpanSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/fpm/PrefixSpanSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/fpm/PrefixSpanSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
similarity index 70%
rename from mllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
index 67c64f762b2..76dd5ea4211 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
+++ b/mllib/core/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala
@@ -24,21 +24,6 @@ import org.apache.spark.sql.types._
class VectorUDTSuite extends SparkFunSuite {
- test("preloaded VectorUDT") {
- val dv1 = Vectors.dense(Array.empty[Double])
- val dv2 = Vectors.dense(1.0, 2.0)
- val sv1 = Vectors.sparse(2, Array.empty, Array.empty)
- val sv2 = Vectors.sparse(2, Array(1), Array(2.0))
-
- for (v <- Seq(dv1, dv2, sv1, sv2)) {
- val udt = UDTRegistration.getUDTFor(v.getClass.getName).get.getConstructor().newInstance()
- .asInstanceOf[VectorUDT]
- assert(v === udt.deserialize(udt.serialize(v)))
- assert(udt.typeName == "vector")
- assert(udt.simpleString == "vector")
- }
- }
-
test("JavaTypeInference with VectorUDT") {
val (dataType, _) = JavaTypeInference.inferDataType(classOf[LabeledPoint])
assert(dataType.asInstanceOf[StructType].fields.map(_.dataType)
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquaresSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquaresSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquaresSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquaresSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/WeightedLeastSquaresSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/DifferentiableLossAggregatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularizationSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularizationSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularizationSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/loss/DifferentiableRegularizationSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/optim/loss/RDDLossFunctionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/optim/loss/RDDLossFunctionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/optim/loss/RDDLossFunctionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/optim/loss/RDDLossFunctionSuite.scala
diff --git a/mllib/core/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
new file mode 100644
index 00000000000..fe087446455
--- /dev/null
+++ b/mllib/core/src/test/scala/org/apache/spark/ml/param/ParamsSuite.scala
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.param
+
+import java.io.{ByteArrayOutputStream, ObjectOutputStream}
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.util.MyParams
+import org.apache.spark.sql.Dataset
+
+class ParamsSuite extends SparkFunSuite {
+
+ test("Filtering ParamMap") {
+ val params1 = new MyParams("my_params1")
+ val params2 = new MyParams("my_params2")
+ val paramMap = ParamMap(
+ params1.intParam -> 1,
+ params2.intParam -> 1,
+ params1.doubleParam -> 0.2,
+ params2.doubleParam -> 0.2)
+ val filteredParamMap = paramMap.filter(params1)
+
+ assert(filteredParamMap.size === 2)
+ filteredParamMap.toSeq.foreach {
+ case ParamPair(p, _) =>
+ assert(p.parent === params1.uid)
+ }
+
+ // At the previous implementation of ParamMap#filter,
+ // mutable.Map#filterKeys was used internally but
+ // the return type of the method is not serializable (see SI-6654).
+ // Now mutable.Map#filter is used instead of filterKeys and the return type is serializable.
+ // So let's ensure serializability.
+ val objOut = new ObjectOutputStream(new ByteArrayOutputStream())
+ objOut.writeObject(filteredParamMap)
+ }
+}
+
+object ParamsSuite extends SparkFunSuite {
+
+ /**
+ * Checks common requirements for `Params.params`:
+ * - params are ordered by names
+ * - param parent has the same UID as the object's UID
+ * - param name is the same as the param method name
+ * - obj.copy should return the same type as the obj
+ */
+ def checkParams(obj: Params): Unit = {
+ val clazz = obj.getClass
+
+ val params = obj.params
+ val paramNames = params.map(_.name)
+ require(paramNames === paramNames.sorted, "params must be ordered by names")
+ params.foreach { p =>
+ assert(p.parent === obj.uid)
+ assert(obj.getParam(p.name) === p)
+ // TODO: Check that setters return self, which needs special handling for generic types.
+ }
+
+ val copyMethod = clazz.getMethod("copy", classOf[ParamMap])
+ val copyReturnType = copyMethod.getReturnType
+ require(copyReturnType === obj.getClass,
+ s"${clazz.getName}.copy should return ${clazz.getName} instead of ${copyReturnType.getName}.")
+ }
+
+ /**
+ * Checks that the class throws an exception in case multiple exclusive params are set.
+ * The params to be checked are passed as arguments with their value.
+ */
+ def testExclusiveParams(
+ model: Params,
+ dataset: Dataset[_],
+ paramsAndValues: (String, Any)*): Unit = {
+ val m = model.copy(ParamMap.empty)
+ paramsAndValues.foreach { case (paramName, paramValue) =>
+ m.set(m.getParam(paramName), paramValue)
+ }
+ intercept[IllegalArgumentException] {
+ m match {
+ case t: Transformer => t.transform(dataset)
+ case e: Estimator[_] => e.fit(dataset)
+ }
+ }
+ }
+}
diff --git a/mllib/src/test/scala/org/apache/spark/ml/python/MLSerDeSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/python/MLSerDeSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/python/MLSerDeSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/python/MLSerDeSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/r/RWrapperUtilsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/r/RWrapperUtilsSuite.scala
similarity index 94%
rename from mllib/src/test/scala/org/apache/spark/ml/r/RWrapperUtilsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/r/RWrapperUtilsSuite.scala
index 27b03918d95..f77c9e95f9f 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/r/RWrapperUtilsSuite.scala
+++ b/mllib/core/src/test/scala/org/apache/spark/ml/r/RWrapperUtilsSuite.scala
@@ -25,7 +25,7 @@ class RWrapperUtilsSuite extends SparkFunSuite with MLlibTestSparkContext {
test("avoid libsvm data column name conflicting") {
val rFormula = new RFormula().setFormula("label ~ features")
- val data = spark.read.format("libsvm").load("../data/mllib/sample_libsvm_data.txt")
+ val data = spark.read.format("libsvm").load("../../data/mllib/sample_libsvm_data.txt")
// if not checking column name, then IllegalArgumentException
intercept[IllegalArgumentException] {
diff --git a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/recommendation/CollectTopKSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/recommendation/CollectTopKSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/recommendation/CollectTopKSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/recommendation/CollectTopKSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/DecisionTreeRegressorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/DecisionTreeRegressorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/DecisionTreeRegressorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/DecisionTreeRegressorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/FMRegressorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/FMRegressorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/FMRegressorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/FMRegressorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/GBTRegressorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/GBTRegressorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/GBTRegressorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/GBTRegressorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/IsotonicRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/IsotonicRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/IsotonicRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/IsotonicRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/ANOVATestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/ANOVATestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/ANOVATestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/ANOVATestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/CorrelationSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/CorrelationSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/CorrelationSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/CorrelationSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/FValueTestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/FValueTestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/FValueTestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/FValueTestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/MultiClassSummarizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/MultiClassSummarizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/MultiClassSummarizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/MultiClassSummarizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/BaggedPointSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/BaggedPointSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tree/impl/BaggedPointSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/BaggedPointSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/GradientBoostedTreesSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/GradientBoostedTreesSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tree/impl/GradientBoostedTreesSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/GradientBoostedTreesSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/RandomForestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreePointSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/TreePointSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreePointSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/TreePointSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tuning/ParamGridBuilderSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tuning/ParamGridBuilderSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tuning/ParamGridBuilderSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tuning/ParamGridBuilderSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tuning/TrainValidationSplitSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tuning/TrainValidationSplitSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tuning/TrainValidationSplitSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tuning/TrainValidationSplitSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tuning/ValidatorParamsSuiteHelpers.scala b/mllib/core/src/test/scala/org/apache/spark/ml/tuning/ValidatorParamsSuiteHelpers.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/tuning/ValidatorParamsSuiteHelpers.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/tuning/ValidatorParamsSuiteHelpers.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/MLTest.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/MLTest.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/MLTestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/MLTestingUtils.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/MLTestingUtils.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/MLTestingUtils.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/MLTestingUtils.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/PMMLReadWriteTest.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/PMMLReadWriteTest.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/PMMLReadWriteTest.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/PMMLReadWriteTest.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/PMMLUtils.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/PMMLUtils.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/PMMLUtils.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/PMMLUtils.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/ReadWriteSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/ReadWriteSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/ReadWriteSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/ReadWriteSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/StopwatchSuite.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/StopwatchSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/StopwatchSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/StopwatchSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala b/mllib/core/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala
rename to mllib/core/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/api/python/PythonMLLibAPISuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/api/python/PythonMLLibAPISuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/api/python/PythonMLLibAPISuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/api/python/PythonMLLibAPISuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/classification/NaiveBayesSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/classification/NaiveBayesSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/classification/NaiveBayesSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/classification/NaiveBayesSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/classification/SVMSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/classification/SVMSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/classification/SVMSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/classification/SVMSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/BisectingKMeansSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/BisectingKMeansSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/BisectingKMeansSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/BisectingKMeansSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/DistanceMeasureSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/DistanceMeasureSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/DistanceMeasureSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/DistanceMeasureSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/GaussianMixtureSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/GaussianMixtureSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/GaussianMixtureSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/GaussianMixtureSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/PowerIterationClusteringSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/PowerIterationClusteringSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/PowerIterationClusteringSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/PowerIterationClusteringSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/clustering/StreamingKMeansSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/clustering/StreamingKMeansSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/clustering/StreamingKMeansSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/clustering/StreamingKMeansSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/evaluation/AreaUnderCurveSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/AreaUnderCurveSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/evaluation/AreaUnderCurveSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/AreaUnderCurveSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetricsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetricsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetricsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetricsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/evaluation/MultilabelMetricsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/MultilabelMetricsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/evaluation/MultilabelMetricsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/MultilabelMetricsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/evaluation/RegressionMetricsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/RegressionMetricsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/evaluation/RegressionMetricsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/evaluation/RegressionMetricsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/ChiSqSelectorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/ChiSqSelectorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/ChiSqSelectorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/ChiSqSelectorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/ElementwiseProductSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/ElementwiseProductSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/ElementwiseProductSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/ElementwiseProductSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/HashingTFSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/HashingTFSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/HashingTFSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/HashingTFSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/NormalizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/NormalizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/NormalizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/NormalizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/PCASuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/PCASuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/PCASuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/PCASuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/StandardScalerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/StandardScalerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/StandardScalerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/StandardScalerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/fpm/AssociationRulesSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/fpm/AssociationRulesSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/fpm/AssociationRulesSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/fpm/AssociationRulesSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/fpm/FPGrowthSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/fpm/FPGrowthSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/fpm/FPGrowthSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/fpm/FPGrowthSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/fpm/FPTreeSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/fpm/FPTreeSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/fpm/FPTreeSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/fpm/FPTreeSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/fpm/PrefixSpanSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/fpm/PrefixSpanSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/fpm/PrefixSpanSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/fpm/PrefixSpanSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/BLASSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/BLASSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/BLASSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/BLASSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/BreezeMatrixConversionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/BreezeMatrixConversionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/BreezeMatrixConversionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/BreezeMatrixConversionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/BreezeVectorConversionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/BreezeVectorConversionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/BreezeVectorConversionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/BreezeVectorConversionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrixSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrixSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrixSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrixSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrixSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrixSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrixSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/CoordinateMatrixSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrixSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrixSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrixSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/IndexedRowMatrixSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/optimization/GradientDescentSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/optimization/GradientDescentSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/optimization/GradientDescentSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/optimization/GradientDescentSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/optimization/LBFGSSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/optimization/LBFGSSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/optimization/LBFGSSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/optimization/LBFGSSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/optimization/NNLSSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/optimization/NNLSSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/optimization/NNLSSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/optimization/NNLSSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExportSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExportSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExportSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExportSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExportSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExportSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExportSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExportSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExportSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExportSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExportSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/KMeansPMMLModelExportSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactorySuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactorySuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactorySuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/pmml/export/PMMLModelExportFactorySuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/random/RandomDataGeneratorSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/random/RandomDataGeneratorSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/random/RandomDataGeneratorSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/random/RandomDataGeneratorSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/random/RandomRDDsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/random/RandomRDDsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/random/RandomRDDsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/random/RandomRDDsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctionsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctionsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctionsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctionsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/rdd/RDDFunctionsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/rdd/RDDFunctionsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/rdd/RDDFunctionsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/rdd/RDDFunctionsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/recommendation/ALSSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/recommendation/ALSSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/recommendation/ALSSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/recommendation/ALSSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModelSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModelSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModelSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModelSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/regression/IsotonicRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/regression/IsotonicRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/regression/IsotonicRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/regression/IsotonicRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/regression/LabeledPointSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/regression/LabeledPointSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/regression/LabeledPointSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/regression/LabeledPointSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/regression/LassoSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/regression/LassoSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/regression/LassoSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/regression/LassoSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/regression/LinearRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/regression/LinearRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/regression/LinearRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/regression/LinearRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/regression/RidgeRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/regression/RidgeRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/regression/RidgeRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/regression/RidgeRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/stat/CorrelationSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/stat/CorrelationSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/stat/CorrelationSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/stat/CorrelationSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/stat/HypothesisTestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/stat/HypothesisTestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/stat/HypothesisTestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/stat/HypothesisTestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/stat/KernelDensitySuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/stat/KernelDensitySuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/stat/KernelDensitySuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/stat/KernelDensitySuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizerSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizerSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizerSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizerSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/stat/StreamingTestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/stat/StreamingTestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/stat/StreamingTestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/stat/StreamingTestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussianSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussianSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussianSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussianSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/EnsembleTestHelper.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/tree/EnsembleTestHelper.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/tree/EnsembleTestHelper.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/tree/EnsembleTestHelper.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/ImpuritySuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/tree/ImpuritySuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/tree/ImpuritySuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/tree/ImpuritySuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/RandomForestSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/tree/RandomForestSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/tree/RandomForestSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/tree/RandomForestSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/MLlibTestSparkContext.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/util/MLlibTestSparkContext.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/util/MLlibTestSparkContext.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/util/MLlibTestSparkContext.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/NumericParserSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/util/NumericParserSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/util/NumericParserSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/util/NumericParserSuite.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/TestingUtils.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/util/TestingUtils.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/util/TestingUtils.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/util/TestingUtils.scala
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/TestingUtilsSuite.scala b/mllib/core/src/test/scala/org/apache/spark/mllib/util/TestingUtilsSuite.scala
similarity index 100%
rename from mllib/src/test/scala/org/apache/spark/mllib/util/TestingUtilsSuite.scala
rename to mllib/core/src/test/scala/org/apache/spark/mllib/util/TestingUtilsSuite.scala
diff --git a/mllib-local/benchmarks/BLASBenchmark-jdk11-results.txt b/mllib/local/benchmarks/BLASBenchmark-jdk11-results.txt
similarity index 100%
rename from mllib-local/benchmarks/BLASBenchmark-jdk11-results.txt
rename to mllib/local/benchmarks/BLASBenchmark-jdk11-results.txt
diff --git a/mllib-local/benchmarks/BLASBenchmark-jdk17-results.txt b/mllib/local/benchmarks/BLASBenchmark-jdk17-results.txt
similarity index 100%
rename from mllib-local/benchmarks/BLASBenchmark-jdk17-results.txt
rename to mllib/local/benchmarks/BLASBenchmark-jdk17-results.txt
diff --git a/mllib-local/benchmarks/BLASBenchmark-results.txt b/mllib/local/benchmarks/BLASBenchmark-results.txt
similarity index 100%
rename from mllib-local/benchmarks/BLASBenchmark-results.txt
rename to mllib/local/benchmarks/BLASBenchmark-results.txt
diff --git a/mllib-local/pom.xml b/mllib/local/pom.xml
similarity index 98%
rename from mllib-local/pom.xml
rename to mllib/local/pom.xml
index 00c16a8b6a5..8146658c579 100644
--- a/mllib-local/pom.xml
+++ b/mllib/local/pom.xml
@@ -22,7 +22,7 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.5.0-SNAPSHOT</version>
- <relativePath>../pom.xml</relativePath>
+ <relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>spark-mllib-local_2.12</artifactId>
diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/impl/Utils.scala b/mllib/local/src/main/scala/org/apache/spark/ml/impl/Utils.scala
similarity index 100%
rename from mllib-local/src/main/scala/org/apache/spark/ml/impl/Utils.scala
rename to mllib/local/src/main/scala/org/apache/spark/ml/impl/Utils.scala
diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala b/mllib/local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala
similarity index 100%
rename from mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala
rename to mllib/local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala
diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala b/mllib/local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
similarity index 100%
rename from mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
rename to mllib/local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala b/mllib/local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala
similarity index 100%
rename from mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala
rename to mllib/local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala
diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala b/mllib/local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala
similarity index 100%
rename from mllib-local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala
rename to mllib/local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/SparkMLFunSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/SparkMLFunSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/SparkMLFunSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/SparkMLFunSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/impl/UtilsSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/impl/UtilsSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/impl/UtilsSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/impl/UtilsSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/BLASBenchmark.scala b/mllib/local/src/test/scala/org/apache/spark/ml/linalg/BLASBenchmark.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/linalg/BLASBenchmark.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/linalg/BLASBenchmark.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/BLASSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/linalg/BLASSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/linalg/BLASSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/linalg/BLASSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/BreezeMatrixConversionSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/linalg/BreezeMatrixConversionSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/linalg/BreezeMatrixConversionSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/linalg/BreezeMatrixConversionSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/BreezeVectorConversionSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/linalg/BreezeVectorConversionSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/linalg/BreezeVectorConversionSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/linalg/BreezeVectorConversionSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/VectorsSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/linalg/VectorsSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/linalg/VectorsSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/linalg/VectorsSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussianSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussianSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussianSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussianSuite.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/util/TestingUtils.scala b/mllib/local/src/test/scala/org/apache/spark/ml/util/TestingUtils.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/util/TestingUtils.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/util/TestingUtils.scala
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/util/TestingUtilsSuite.scala b/mllib/local/src/test/scala/org/apache/spark/ml/util/TestingUtilsSuite.scala
similarity index 100%
rename from mllib-local/src/test/scala/org/apache/spark/ml/util/TestingUtilsSuite.scala
rename to mllib/local/src/test/scala/org/apache/spark/ml/util/TestingUtilsSuite.scala
diff --git a/pom.xml b/pom.xml
index 51de315dd8c..3cedeb3e153 100644
--- a/pom.xml
+++ b/pom.xml
@@ -84,8 +84,9 @@
<module>common/tags</module>
<module>core</module>
<module>graphx</module>
- <module>mllib</module>
- <module>mllib-local</module>
+ <module>mllib/core</module>
+ <module>mllib/common</module>
+ <module>mllib/local</module>
<module>tools</module>
<module>streaming</module>
<module>sql/catalyst</module>
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 9741e53452a..05e301ac82f 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -36,6 +36,89 @@ object MimaExcludes {
// Exclude rules for 3.5.x
lazy val v35excludes = v34excludes ++ Seq(
+ // [SPARK-42508][CONNECT][ML] Extract the common .ml classes to mllib-common
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.package"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.package$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.AttributeGroup"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.AttributeGroup$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.AttributeKeys"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.Attribute"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.Attribute$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.AttributeFactory"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.NumericAttribute"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.NumericAttribute$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.NominalAttribute"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.NominalAttribute$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.BinaryAttribute"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.BinaryAttribute$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.UnresolvedAttribute"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.UnresolvedAttribute$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.AttributeType"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.attribute.AttributeType$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.feature.LabeledPoint"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.feature.LabeledPoint$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.linalg.JsonMatrixConverter"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.linalg.JsonVectorConverter"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.linalg.MatrixUDT"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.linalg.VectorUDT"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.linalg.SQLDataTypes"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.linalg.SQLDataTypes$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.Param"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.ParamPair"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.ParamPair$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.JavaParams"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.ParamMap"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.ParamMap$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.ParamValidators"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.ParamValidators$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.DoubleParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.IntParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.FloatParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.LongParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.BooleanParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.IntArrayParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.StringArrayParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.DoubleArrayParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.DoubleArrayArrayParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasAggregationDepth"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasBlockSize"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasCheckpointInterval"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasCollectSubModels"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasDistanceMeasure"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasElasticNetParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasFeaturesCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasFitIntercept"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasHandleInvalid"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasInputCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasInputCols"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasLabelCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasLoss"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasMaxBlockSizeInMB"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasMaxIter"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasNumFeatures"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasOutputCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasOutputCols"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasPredictionCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasProbabilityCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasRawPredictionCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasRegParam"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasRelativeError"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasSeed"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasSolver"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasStandardization"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasStepSize"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasThreshold"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasThresholds"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasTol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasValidationIndicatorCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasVarianceCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.param.shared.HasWeightCol"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.PredictorParams"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.classification.ClassifierParams"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.classification.ProbabilisticClassifierParams"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.util.Identifiable"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.util.Identifiable$"),
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.util.SchemaUtils")
)
// Exclude rules for 3.4.x from 3.3.0
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index ed9cfeb7cd2..f964ba032c5 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -57,9 +57,9 @@ object BuildCommons {
val connectClient = ProjectRef(buildLocation, "connect-client-jvm")
val allProjects@Seq(
- core, graphx, mllib, mllibLocal, repl, networkCommon, networkShuffle, launcher, unsafe, tags, sketch, kvstore, _*
+ core, graphx, mllib, mllibLocal, mllibCommon, repl, networkCommon, networkShuffle, launcher, unsafe, tags, sketch, kvstore, _*
) = Seq(
- "core", "graphx", "mllib", "mllib-local", "repl", "network-common", "network-shuffle", "launcher", "unsafe",
+ "core", "graphx", "mllib", "mllib-local", "mllib-common", "repl", "network-common", "network-shuffle", "launcher", "unsafe",
"tags", "sketch", "kvstore"
).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects ++ Seq(connectCommon, connect, connectClient)
@@ -403,7 +403,8 @@ object SparkBuild extends PomBuild {
val mimaProjects = allProjects.filterNot { x =>
Seq(
spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle, networkYarn,
- unsafe, tags, tokenProviderKafka010, sqlKafka010, connectCommon, connect, connectClient, protobuf
+ unsafe, tags, tokenProviderKafka010, sqlKafka010, connectCommon, connect, connectClient, protobuf,
+ mllibCommon
).contains(x)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org