You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by selvinsource <gi...@git.apache.org> on 2014/11/02 14:40:34 UTC
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
GitHub user selvinsource opened a pull request:
https://github.com/apache/spark/pull/3062
[SPARK-1406] Mllib pmml model export
See PDF attached to the JIRA issue 1406.
The contribution is my original work and I license the work to the project under the project's open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/selvinsource/spark mllib_pmml_model_export_SPARK-1406
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3062.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3062
----
commit a0e36797cd771da1130d0ddcdedaec44b2e6357e
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-12T12:30:49Z
export and pmml export traits
kmeans test implementation
commit 226e1849ef6717fccd07122b13954471ff43cd28
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-18T14:03:45Z
added javadoc and export model type in case there is a need to support
other types of export (not just PMML)
commit 9bc494fa30508d7cd6f2146b22c0dc77750c09d1
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-26T17:50:44Z
added scala suite tests
added saveLocalFile to ModelExport trait
commit 8e71b8d8cc3922e834fe0f21e4f05e773241dad7
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-27T21:58:54Z
kmeans pmml export implementation
commit 1433b1139e510457419f2ffd29195bf29f6e556e
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-29T10:06:07Z
complete suite tests
commit 88414391312e4030e462d7d5b37539211ae60316
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-29T11:25:42Z
adjust scala style in order to compile
commit 07a29bf0a08c146e827302215ded3d20122cf20c
Author: selvinsource <vs...@hotmail.it>
Date: 2014-10-29T11:47:20Z
Update LICENSE
added https://github.com/jpmml/jpmml-model library under BSD-style licenses
the library is used to export the mining model to the PMML format
commit f75b9882b7633e65cbe26ba51401232d7d82e634
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-29T11:51:03Z
Merge remote-tracking branch 'origin/master' into mllib_pmml_model_export_SPARK-1406
commit cd6c07c654141dcd0eb468e09bf70680a318bdb5
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-29T16:57:12Z
fixed scala style to run tests
commit aba5ee1d75b012a17a1db077bdd88c0970310cae
Author: Vincenzo Selvaggio <vs...@hotmail.it>
Date: 2014-10-30T22:18:54Z
fixed cluster export
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-72552402
The R case is slightly different because the `pmml` package is maintained separately from other packages. @srowen , are you okay with the interface I proposed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61419907
[Test build #22763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22763/consoleFull) for PR 3062 at commit [`aba5ee1`](https://github.com/apache/spark/commit/aba5ee1d75b012a17a1db077bdd88c0970310cae).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait ModelExport `
* `class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport`
* `trait PMMLModelExport extends ModelExport`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94758704
@mengxr
FastInfoset-1.2.12.jar -> http://en.wikipedia.org/wiki/Fast_Infoset -> Apache License 2.0
istack-commons-runtime-2.16.jar -> https://istack-commons.java.net/ -> CDDL v1.1
jaxb* -> https://jaxb.java.net/ -> CDDL v1.1
pmml* -> https://github.com/jpmml/jpmml-model -> BSD 3-Clause License
Accordingly to http://www.apache.org/legal/resolved.html CDDL is possible as binary, but please verify too that this is the case.
@vruusmann
I assume that if JPMML-Model is licensed under the BSD 3-Clause License this is compatible with CDDL v1.1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97141054
[Test build #31147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31147/consoleFull) for PR 3062 at commit [`30165c4`](https://github.com/apache/spark/commit/30165c4caaea77b5b91ce2bef0ca94cb0454f527).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by vruusmann <gi...@git.apache.org>.
Github user vruusmann commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94781590
@selvinsource The benefits of upgrading the `pmml-model` dependency are more obvious if you are in the business of consuming PMML documents (eg. speed and memory usage improvements). But I still recommend you to make this move.
The upgrade from 1.1.7 to 1.1.13 should be very easy. The upgrade from 1.1.13 to 1.1.14 may require touching Java source code here and there (eg. rearranging the arguments of value constructors, correcting the case of selected method names).
The upgrade will change the list of your transitive dependencies. So, you should probably to the upgrade first, and edit `LICENSE` and `NOTICE` files second.
Moreover, I think that after the upgrade it will be safe to exclude `FastInfoset` and `istack-commons-runtime` dependencies from the build (using Maven's dependency exclusion mechanism) without affecting the PMML exporter functionality. These dependencies take care of managing SAX Locator information when unmarshalling. But since you are creating new PMML documents in memory from scratch, then they are not needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97555605
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73423503
[Test build #27048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27048/consoleFull) for PR 3062 at commit [`7a949d0`](https://github.com/apache/spark/commit/7a949d0b3f5b397f4df4fcd70f3658cc66e8f04e).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-143408820
@tvmanikandan
we are adding decision tree PMML support for multiclass classification, see https://issues.apache.org/jira/browse/SPARK-8542.
Linear SVMs supports only binary classification, while logistic regression supports both binary and multiclass classification problems. If you would like logistic regression multiclass classification PMML export and the decision tree is not enough, please create a Jira issue under the https://issues.apache.org/jira/browse/SPARK-8545 PMML improvement umbrella.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94287249
@mengxr I aligned the branch with the master and resolved a conflict so that it should be easier for you to review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73422812
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27046/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-72567721
@srowen Yes, the implementation would live under the `mllib.pmml` package and we add `toPMML` to models that we support. For the return type, exposing third-party APIs is not recommended in Spark. We used Guava's Optional, which turned out to be a bad choice for maintenance. There are pros/cons for other types.
1. String. Easy to save/load, but it may not be able to handle super large XML.
2. `Result` from `javax.xml`. Officially supported by Java, but most users don't know how to use it.
So it really depends on how big the model is and whether it is necessary to output XML as a stream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #3062: [SPARK-1406] Mllib pmml model export
Posted by vruusmann <gi...@git.apache.org>.
Github user vruusmann commented on the issue:
https://github.com/apache/spark/pull/3062
@manugarri You can export fitted pipeline models to PMML using the [JPMML-SparkML-Package](https://github.com/jpmml/jpmml-sparkml-package) Apache Spark Package. There's a worked-out PySpark example in its README.md file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-66888410
[Test build #24437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24437/consoleFull) for PR 3062 at commit [`8fe12bb`](https://github.com/apache/spark/commit/8fe12bb58358c0d08ede99ed9520e386aedf5110).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97636493
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-75657218
Sounds good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-75626745
Yeah that makes sense, I don't think the xml would be that big to require multiple partitions:
sc.parallelize(Array(pmml),1).saveAsTextFile(path)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94704255
[Test build #30646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30646/consoleFull) for PR 3062 at commit [`e2313df`](https://github.com/apache/spark/commit/e2313df33560bfd1e8377ff33be8a67d5ad0922a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `FastInfoset-1.2.12.jar`
* `istack-commons-runtime-2.16.jar`
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `jsr173_api-1.0.jar`
* `pmml-model-1.1.7.jar`
* `pmml-schema-1.1.7.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95095624
@selvinsource it's not necessary to update the version. I think it's there since I auto-generated these lists from a Maven report at the outset.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94561695
@selvinsource I sent you a PR at https://github.com/selvinsource/spark/pull/1 to update the code style.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94292802
[Test build #30564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30564/consoleFull) for PR 3062 at commit [`1676e15`](https://github.com/apache/spark/commit/1676e15ff5bec346466a095a593efdd324f6b515).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96778091
@selvinsource Thanks for testing! If the threshold is set to none, I think we should use 0.5 for logistic regression and 0.0 for SVM, maybe with warning log messages. Your proposal on how to set the intercept for NO in logistic regression looks good to me. Please remember to handle the extreme case (when threshold = 1.0 or 0.0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73422771
[Test build #27046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27046/consoleFull) for PR 3062 at commit [`f46c75c`](https://github.com/apache/spark/commit/f46c75cdbea8170c332026ffd05d5ca1667d50c6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-93664878
I'm going to make a pass this week. The target is 1.4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97570247
[Test build #31313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31313/consoleFull) for PR 3062 at commit [`852aac6`](https://github.com/apache/spark/commit/852aac6149d4d0c932ff34e959d148419b369983).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
* `spark-unsafe_2.10-1.4.0-SNAPSHOT.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64947311
[Test build #23946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23946/consoleFull) for PR 3062 at commit [`1faf985`](https://github.com/apache/spark/commit/1faf985b82afec0f5bf0a8dc9cd4f387390aa487).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97636485
[Test build #31357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31357/consoleFull) for PR 3062 at commit [`852aac6`](https://github.com/apache/spark/commit/852aac6149d4d0c932ff34e959d148419b369983).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19720383
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
+
+ var miningSchema = new MiningSchema()
+
+ var comparisonMeasure = new ComparisonMeasure()
+ .withKind(Kind.DISTANCE)
+ .withMeasure(new SquaredEuclidean()
+ );
+
+ var clusteringModel = new ClusteringModel(miningSchema, comparisonMeasure,
+ MiningFunctionType.CLUSTERING, ModelClass.CENTER_BASED, model.clusterCenters.length)
+ .withModelName("k-means");
+
+ for ( i <- 0 to (clusterCenter.size - 1)) {
+ fields(i) = FieldName.create("field_" + i)
--- End diff --
This is a notable issue to discuss -- the MLlib models don't have any indication of what the fields are. Although you can make up a "field_X" name for them it kind of makes the resulting model unusable. I think this may require later changes that may be coming in MLlib where MLlib might have more of a notion of schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61416747
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by damiencarol <gi...@git.apache.org>.
Github user damiencarol commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-93572357
Any news on this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61802606
[Test build #22938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22938/consoleFull) for PR 3062 at commit [`e29dfb9`](https://github.com/apache/spark/commit/e29dfb99ffecb3cd78e5f2a1357b2244bc9c9737).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by vruusmann <gi...@git.apache.org>.
Github user vruusmann commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95080549
@selvinsource @mengxr I created an small project https://github.com/vruusmann/jpmml-test in order to demonstrate how it's possible to reduce the number of dependencies.
The idea is that you should keep JAXB dependencies (scope: `compile`) when building for Java 1.6, but you can safely exclude them (scope: `provided`) when building for Java 1.7 or newer.
Here's the output of `mvn dependency:tree` command for Java 1.6 (the size of the resulting uber-JAR file is 1.8 MB):
```
[INFO] org.jpmml:pmml-test:jar:1.0-SNAPSHOT
[INFO] +- org.jpmml:pmml-model:jar:1.1.15:compile
[INFO] | +- org.jpmml:pmml-agent:jar:1.1.15:compile
[INFO] | \- org.jpmml:pmml-schema:jar:1.1.15:compile
[INFO] \- com.sun.xml.bind:jaxb-impl:jar:2.2.7:compile
[INFO] \- com.sun.xml.bind:jaxb-core:jar:2.2.7:compile
[INFO] \- javax.xml.bind:jaxb-api:jar:2.2.7:compile
```
Here's the same for Java 1.7 (the size of the uber-JAR file has shrunk by 1.2 MB to 0.6 MB; all the remaining dependencies `pmml-model`, `pmml-schema` and `pmml-agent` are covered by the BSD 3-Clause License):
```
[INFO] org.jpmml:pmml-test:jar:1.0-SNAPSHOT
[INFO] +- org.jpmml:pmml-model:jar:1.1.15:compile
[INFO] | +- org.jpmml:pmml-agent:jar:1.1.15:compile
[INFO] | \- org.jpmml:pmml-schema:jar:1.1.15:compile
[INFO] \- com.sun.xml.bind:jaxb-impl:jar:2.2.7:provided
[INFO] +- com.sun.xml.bind:jaxb-core:jar:2.2.7:provided
[INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.7:provided
[INFO] | \- com.sun.istack:istack-commons-runtime:jar:2.16:provided
[INFO] \- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.12:provided
[INFO] \- javax.xml.bind:jsr173_api:jar:1.0:provided
```
JAXB dependencies were made part of Java runtime sometimes in the middle of Java 1.6 (for example, 1.6_u01 does not have them, whereas 1.6_u45 does). So, it's probably possible to fine-tune the activation of the Maven build profile some more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95086612
@srowen
https://github.com/apache/spark/blob/master/NOTICE already has a reference to JAXB:
(CDDL 1.1) (GPL2 w/ CPE) JAXB API bundle for GlassFish V3 (javax.xml.bind:jaxb-api:2.2.2 - https://jaxb.dev.java.net/)
(CDDL 1.1) (GPL2 w/ CPE) JAXB RI (com.sun.xml.bind:jaxb-impl:2.2.3-1 - http://jaxb.java.net/)
Is it just a matter of updating the NOTICE file with the new version 2.2.7?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97614086
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94288556
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30562/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61416751
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64947314
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23946/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-72540267
@srowen @selvinsource From user perspective, this is the code they need to export a model to PMML:
~~~
import org.apache.spark.mllib.export.ModelExporter
val kmeansModel = ...
ModelExporter.toPMML(kmeansModel, "/path")
~~~
They need to remember two things: 1) `ModelExporter`'s name and its package, 2) whether a model is exportable to PMML or not because `toPMML` takes `Any`. As a user, I would prefer the following
~~~
val kmeansModel = ...
val pmml = kmeansModel.toPMML
~~~
We can add a trait called `PMMLExportable`, which contains `toPMML`. (We need to discuss the return type of `toPMML`.) So users know whether a model can be exported to PMML or not. For the implementation, I agree with @srowen that `.pmml` package should be sufficient because there is no other exchangeable formats for predictive models. The real export code for each model should live under this package as private and `toPMML` in each model becomes a simple wrapper.
Due to lacking feature attributes, I'd like to mark everything as private for now until we add ML attributes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-74729850
@selvinsource I like the output options which write to disk, but what about supporting writing to distributed file systems? It's more common in Spark to use URIs instead of local paths, but it might be reasonable to make an exception for PMML since it might often be used for small models. What are your thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19866949
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import scala.beans.BeanProperty
+
+import org.dmg.pmml.Application
+import org.dmg.pmml.Header
+import org.dmg.pmml.PMML
+import org.dmg.pmml.Timestamp
+
+import org.apache.spark.mllib.export.ModelExport
+
+private[mllib] trait PMMLModelExport extends ModelExport{
+
+ /**
+ * Holder of the exported model in PMML format
+ */
+ @BeanProperty
+ val pmml: PMML = new PMML();
+
+ setHeader(pmml);
+
+ private def setHeader(pmml : PMML): Unit = {
+ val version = getClass().getPackage().getImplementationVersion()
+ val app = new Application().withName("Apache Spark MLlib").withVersion(version)
+ val timestamp = new Timestamp()
+ .withContent(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date()))
+ val header = new Header()
+ .withCopyright("Apache Spark MLlib")
+ .withApplication(app)
+ .withTimestamp(timestamp);
+ pmml.setHeader(header);
+ pmml.setVersion("4.2")
--- End diff --
4.2.1 is technically the latest version... although I don't know that we're using any features that require that version, I'd default to specifying the current latest version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73422811
[Test build #27046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27046/consoleFull) for PR 3062 at commit [`f46c75c`](https://github.com/apache/spark/commit/f46c75cdbea8170c332026ffd05d5ca1667d50c6).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable with PMMLExportable `
* `trait PMMLExportable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94751364
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30661/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94796320
+1 to updating to the latest 1.1.x JPMML. I have had good experiences with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97556296
[Test build #31313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31313/consoleFull) for PR 3062 at commit [`852aac6`](https://github.com/apache/spark/commit/852aac6149d4d0c932ff34e959d148419b369983).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61407049
https://issues.apache.org/jira/browse/SPARK-1406
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-75478987
I see, so the SparkContext makes it clear it's being saved to a DFS. That sounds great to me. (Would you want to enforce numPartitions = 1 so that it is stored on a single partition?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95067411
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30733/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r29271367
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.pmml.export
+
+import scala.{Array => SArray}
+
+import org.dmg.pmml._
+
+import org.apache.spark.mllib.regression.GeneralizedLinearModel
+
+/**
+ * PMML Model Export for GeneralizedLinearModel class with binary ClassificationModel
+ */
+private[mllib] class BinaryClassificationPMMLModelExport(
+ model : GeneralizedLinearModel,
+ description : String,
+ normalizationMethod : RegressionNormalizationMethodType,
+ threshold: Double)
+ extends PMMLModelExport {
+
+ populateBinaryClassificationPMML()
+
+ /**
+ * Export the input LogisticRegressionModel or SVMModel to PMML format.
+ */
+ private def populateBinaryClassificationPMML(): Unit = {
+ pmml.getHeader.setDescription(description)
+
+ if (model.weights.size > 0) {
+ val fields = new SArray[FieldName](model.weights.size)
+ val dataDictionary = new DataDictionary
+ val miningSchema = new MiningSchema
+ val regressionTableYES = new RegressionTable(model.intercept).withTargetCategory("1")
+ var interceptNO = threshold
+ if (RegressionNormalizationMethodType.LOGIT == normalizationMethod) {
+ if (threshold <= 0)
+ interceptNO = -1000
--- End diff --
Should be `Double.MinValue`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97601028
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94288552
[Test build #30562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30562/consoleFull) for PR 3062 at commit [`b8823b0`](https://github.com/apache/spark/commit/b8823b022129ad49836b90745a8aa397387c60cf).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Saveable with Serializable with PMMLExportable `
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `FastInfoset-1.2.12.jar`
* `istack-commons-runtime-2.16.jar`
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `jsr173_api-1.0.jar`
* `pmml-model-1.1.7.jar`
* `pmml-schema-1.1.7.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19870786
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import scala.beans.BeanProperty
+
+import org.dmg.pmml.Application
+import org.dmg.pmml.Header
+import org.dmg.pmml.PMML
+import org.dmg.pmml.Timestamp
+
+import org.apache.spark.mllib.export.ModelExport
+
+private[mllib] trait PMMLModelExport extends ModelExport{
+
+ /**
+ * Holder of the exported model in PMML format
+ */
+ @BeanProperty
+ val pmml: PMML = new PMML();
+
+ setHeader(pmml);
+
+ private def setHeader(pmml : PMML): Unit = {
+ val version = getClass().getPackage().getImplementationVersion()
+ val app = new Application().withName("Apache Spark MLlib").withVersion(version)
+ val timestamp = new Timestamp()
+ .withContent(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date()))
+ val header = new Header()
+ .withCopyright("Apache Spark MLlib")
--- End diff --
copyright removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19866926
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import scala.beans.BeanProperty
+
+import org.dmg.pmml.Application
+import org.dmg.pmml.Header
+import org.dmg.pmml.PMML
+import org.dmg.pmml.Timestamp
+
+import org.apache.spark.mllib.export.ModelExport
+
+private[mllib] trait PMMLModelExport extends ModelExport{
+
+ /**
+ * Holder of the exported model in PMML format
+ */
+ @BeanProperty
+ val pmml: PMML = new PMML();
+
+ setHeader(pmml);
+
+ private def setHeader(pmml : PMML): Unit = {
+ val version = getClass().getPackage().getImplementationVersion()
+ val app = new Application().withName("Apache Spark MLlib").withVersion(version)
+ val timestamp = new Timestamp()
+ .withContent(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date()))
+ val header = new Header()
+ .withCopyright("Apache Spark MLlib")
--- End diff --
The header should not have a copyright. Spark MLlib can't claim copyright in models generated by this tool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-76622797
[Test build #28149 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28149/consoleFull) for PR 3062 at commit [`b25bbf7`](https://github.com/apache/spark/commit/b25bbf7ae7eb92541f59f5cfe7495280d60fca43).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96516889
@mengxr for SVM, I manually tried what you suggested and it looks good.
I loaded the example below in JPMML and evaluated it as Classification map, indeed the intercept on the NO category acts as threshold when `normalizationMethod = none`.
Here the example:
<code>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<PMML xmlns="http://www.dmg.org/PMML-4_2">
<Header description="linear SVM: if predicted value > 0, the outcome is positive, or negative otherwise">
<Application name="Apache Spark MLlib" version="1.4.0-SNAPSHOT"/>
<Timestamp>2015-04-27T06:58:22</Timestamp>
</Header>
<DataDictionary numberOfFields="10">
<DataField name="field_0" optype="continuous" dataType="double"/>
<DataField name="field_1" optype="continuous" dataType="double"/>
<DataField name="field_2" optype="continuous" dataType="double"/>
<DataField name="field_3" optype="continuous" dataType="double"/>
<DataField name="field_4" optype="continuous" dataType="double"/>
<DataField name="field_5" optype="continuous" dataType="double"/>
<DataField name="field_6" optype="continuous" dataType="double"/>
<DataField name="field_7" optype="continuous" dataType="double"/>
<DataField name="field_8" optype="continuous" dataType="double"/>
<DataField name="target" optype="categorical" dataType="string"/>
</DataDictionary>
<RegressionModel modelName="linear SVM: if predicted value > 0, the outcome is positive, or negative otherwise" functionName="classification" normalizationMethod="none">
<MiningSchema>
<MiningField name="field_0" usageType="active"/>
<MiningField name="field_1" usageType="active"/>
<MiningField name="field_2" usageType="active"/>
<MiningField name="field_3" usageType="active"/>
<MiningField name="field_4" usageType="active"/>
<MiningField name="field_5" usageType="active"/>
<MiningField name="field_6" usageType="active"/>
<MiningField name="field_7" usageType="active"/>
<MiningField name="field_8" usageType="active"/>
<MiningField name="target" usageType="target"/>
</MiningSchema>
<RegressionTable intercept="-1.2973802920137774" targetCategory="1">
<NumericPredictor name="field_0" coefficient="-0.0818303650185629"/>
<NumericPredictor name="field_1" coefficient="0.5609579878511747"/>
<NumericPredictor name="field_2" coefficient="0.1382792114252377"/>
<NumericPredictor name="field_3" coefficient="0.07497131265977852"/>
<NumericPredictor name="field_4" coefficient="-0.47760356523751296"/>
<NumericPredictor name="field_5" coefficient="0.3817837986572615"/>
<NumericPredictor name="field_6" coefficient="-0.23753782335208481"/>
<NumericPredictor name="field_7" coefficient="0.2548602390316011"/>
<NumericPredictor name="field_8" coefficient="-0.10271528637619945"/>
</RegressionTable>
<RegressionTable intercept="0.0" targetCategory="0"/>
</RegressionModel>
</PMML>
</code>
However, I noticed that if the SVM model threshold is set to None, it simply displays the margin (which is how it is implemented now in the pmml exporter).
My question is, should we support both? If `threshold = None`, export as regression (like it is implemented now), if `threshold <> None`, export as binary classification (as you suggested). What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-70325906
@selvinsource This will be great to get PMML support! I'll try to give some feedback soon.
Quick question: Do you know if there is a freely available PMML validator we could use for checking syntax? Or is JPMML checked thoroughly already? (The AWS and iGoogle links for the Zementis validator seem to be broken now.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96770135
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-70366247
@jkbradley
I use JPMML to verify the exported model produces the same results, here the details of my tests:
https://github.com/selvinsource/spark-pmml-exporter-validator
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61644549
@selvinsource Maybe we can ask if anyone else has an idea? I can think of closed-source formats but I doubt they will be worth supporting. I think there's value in perhaps a thin wrapper on top of all this in order to keep lib-specific classes like `PMML` out of the Spark API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64870646
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23935/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95099568
@srowen that sounds good :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3062
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-72551243
Either approach has its pros and cons and I don't have a strong argument in favor of any of the two.
R, for instance, use a similar approach to what I have done so far and suggested by @srowen
library(pmml)
model0 <- lm(Sepal.Length~., data=iris[,-5])
model <- pmml(model0)
but I understand @mengxr point of view, a user would prefer to just call `.toPMML` on the model.
If we all agree this is the way forward, I can re-organize the code as suggested by @mengxr having the supported models extending the PMMLExportable trait.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97570280
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97570285
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31313/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94286993
[Test build #30562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30562/consoleFull) for PR 3062 at commit [`b8823b0`](https://github.com/apache/spark/commit/b8823b022129ad49836b90745a8aa397387c60cf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61797138
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22931/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97139708
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19768883
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.export.ModelExport
+import java.io.OutputStream
+import org.jpmml.model.JAXBUtil
+import org.dmg.pmml.PMML
+import javax.xml.transform.stream.StreamResult
+import scala.beans.BeanProperty
+import org.dmg.pmml.Application
+import org.dmg.pmml.Timestamp
+import org.dmg.pmml.Header
--- End diff --
ok will fix that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97620507
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94713498
[Test build #30661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30661/consoleFull) for PR 3062 at commit [`a0a55f7`](https://github.com/apache/spark/commit/a0a55f7add73b0fbbb27b285e9316cb23c716aaf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64947819
[Test build #23947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23947/consoleFull) for PR 3062 at commit [`19adf29`](https://github.com/apache/spark/commit/19adf29d0bf2c5ffbdd7ba257296475463dd5eae).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61797130
[Test build #22931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22931/consoleFull) for PR 3062 at commit [`6357b98`](https://github.com/apache/spark/commit/6357b9890e6f4245a2e8a7d0f90dd6d80495e60a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport`
* `trait PMMLModelExport extends ModelExport`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94761183
@selvinsource have a look at our `NOTICE` file to see how to cite these components with their licenses, and `LICENSE` to see what to do with AL2 deps with their own `NOTICE` file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73427140
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27048/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19769621
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.export.ModelExport
+import java.io.OutputStream
+import org.jpmml.model.JAXBUtil
+import org.dmg.pmml.PMML
+import javax.xml.transform.stream.StreamResult
+import scala.beans.BeanProperty
+import org.dmg.pmml.Application
+import org.dmg.pmml.Timestamp
+import org.dmg.pmml.Header
+import java.text.SimpleDateFormat
+import java.util.Date
+
+trait PMMLModelExport extends ModelExport{
+
+ /**
+ * Holder of the exported model in PMML format
+ */
+ @BeanProperty
+ var pmml: PMML = new PMML();
--- End diff --
ok will add 4.2, I missed that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64947297
[Test build #23946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23946/consoleFull) for PR 3062 at commit [`1faf985`](https://github.com/apache/spark/commit/1faf985b82afec0f5bf0a8dc9cd4f387390aa487).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97600567
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by vruusmann <gi...@git.apache.org>.
Github user vruusmann commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94661002
@mengxr Branches 1.0.X and 1.1.X are Java 6. The new branch 1.2.X is Java 7.
The latest Java 6 compatible version is 1.1.15.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19771397
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
+
+ var miningSchema = new MiningSchema()
+
+ var comparisonMeasure = new ComparisonMeasure()
+ .withKind(Kind.DISTANCE)
+ .withMeasure(new SquaredEuclidean()
+ );
+
+ var clusteringModel = new ClusteringModel(miningSchema, comparisonMeasure,
+ MiningFunctionType.CLUSTERING, ModelClass.CENTER_BASED, model.clusterCenters.length)
+ .withModelName("k-means");
+
+ for ( i <- 0 to (clusterCenter.size - 1)) {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94653680
@mengxr I merged your PR, I will review the libraries and let you know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97158970
@selvinsource You can run `dev/lint-scala` on your local to check Scala style.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97761392
@mengxr thanks for merging this PR!
I will look into submitting the evaluator at spark-packages if you think it will be useful, I used it as side project to make sure the exported models were working correctly.
On the documentation, please point me in the right direction and I will be happy to help. We can discuss this on SPARK-7272.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97555803
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73426635
@mengxr @srowen
I created the PMMLExportable trait and moved all the code under the package mllib.pmml.
Supported models implements now this trait and the usage is as you envisaged:
`val pmml = kmeansModel.toPMML()`
I added also extra export options, respectively File and OutputStream:
`kmeansModel.toPMML("/tmp/kmeans.xml")`
`kmeansModel.toPMML(System.out)`
Feel free to review those, I would suggest to keep at least the OutputStream which is well known.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95042057
[Test build #30733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30733/consoleFull) for PR 3062 at commit [`25dce33`](https://github.com/apache/spark/commit/25dce336065e39da8492c6d8379a21ab3733cd45).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97636495
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31357/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97677239
LGTM. Merged into master. Thanks! @selvinsource I created SPARK-7272 for the user guide and assigned it to you. Let me know if you don't have time to do it. Btw, please submit your PMML evaluator code as a Spark package at spark-packages.org. Due to license issues, we cannot include them as example code in the Spark codebase. But PMML model scoring is quite important.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19720365
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
--- End diff --
Why are these 3 vars?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19771997
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
+
+ var miningSchema = new MiningSchema()
+
+ var comparisonMeasure = new ComparisonMeasure()
+ .withKind(Kind.DISTANCE)
+ .withMeasure(new SquaredEuclidean()
+ );
+
+ var clusteringModel = new ClusteringModel(miningSchema, comparisonMeasure,
+ MiningFunctionType.CLUSTERING, ModelClass.CENTER_BASED, model.clusterCenters.length)
+ .withModelName("k-means");
+
+ for ( i <- 0 to (clusterCenter.size - 1)) {
+ fields(i) = FieldName.create("field_" + i)
--- End diff --
However I still find it useful if the end goal of the export is to load it in your own decoupled app, therefore you will know the order of the fields.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96157430
@mengxr
3)
I have done point 3, if it is a multinomial logistic regression the export will cause an IllegalArgumentException. I didn't realize now Logistic supports multiple classes. As you said we can add support for this in a new PR.
1)
For `LogisticRegressionModel` I do use `functionName = "classification"`, see an example of an exported model
https://github.com/selvinsource/spark-pmml-exporter-validator/blob/master/src/main/resources/exported_pmml_models/logisticregression.xml
This can be loaded using JPMML as `ClassificationMap`, see
https://github.com/selvinsource/spark-pmml-exporter-validator/blob/master/src/main/java/org/selvinsource/spark_pmml_exporter_validator/SparkPMMLExporterValidator.java
For `SVMModel`, I did look into exporting it as a PMML classification `RegressionModel` like I did with the `LogisticRegressionModel`, however there isn't a PMML `normalizationMethod` that I can use, they all convert the prediction into probability and `SVMModel` simply compare a scalar to a threshold.
Because of this I added a comment in the exported regression model: "linear SVM: if predicted value > 0, the outcome is positive, or negative otherwise".
2)
I don't see PMML `RegressionModel` supporting a threshold, @vruusmann can you confirm this is the case, or am I missing something :)?
For `SVMModel` export I could add it to the comment, now it is hard coded to 0 but could be "linear SVM: if predicted value > `threshold variable` the outcome is positive, or negative otherwise".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94655050
@selvinsource Thanks! There are some indentation issues in my PR. Could you help fix it?
The latest version of `jpmml-model` requires Java 7 or newer, while Spark needs to support Java 6, 7, and 8. @vruusmann Do you know what is the newest version of `jpmml-model` that supports Java 6?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #3062: [SPARK-1406] Mllib pmml model export
Posted by manugarri <gi...@git.apache.org>.
Github user manugarri commented on the issue:
https://github.com/apache/spark/pull/3062
im not sure if this is the right place to ask, but is there any plan to implement PMML export from pyspark? Cant find anything on the pyspark docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61813097
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22938/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19770371
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
--- End diff --
I guess they don't need to be, I will review var/val usage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95082764
@mengxr code updated, style should be consistent :)!
@vruusmann thanks for your test project, I did the same last night (mvn dependency:tree) to find out what were the transitive dependencies and I explicitely excluded istack-commons-runtime and FastInfoset as you suggested. This leave us with JAXB which I am going to review in terms of license (still need to check the NOTICE and LICENSE files).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19773642
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/ModelExport.scala ---
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export
+
+import java.io.OutputStream
+import java.io.FileOutputStream
+import java.io.File
+
+trait ModelExport {
--- End diff --
I assume you are referring to
JAXBUtil.marshalPMML(pmml, new StreamResult(outputStream));
which takes as param the StreamResult object and for which there are many constructors.
I didn't want to expose the StreamResult as it is xml specific, but if the decision is to only support PMML xml we may add also the StreamResult.
Personally I would still like a simple saveLocalFile(path: String) so that the user doesn't have to think too much and just call:
ModelExportFactory.createModelExport(clusters,ModelExportType.PMML).saveLocalF
ile("/tmp/kmeans.xml") for all models.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97614077
[Test build #31339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31339/consoleFull) for PR 3062 at commit [`852aac6`](https://github.com/apache/spark/commit/852aac6149d4d0c932ff34e959d148419b369983).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
* `spark-unsafe_2.10-1.4.0-SNAPSHOT.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94560020
@selvinsource Could you check whether the following libraries have Apache-compatible license?
* FastInfoset-1.2.12.jar
* istack-commons-runtime-2.16.jar
* jsr173_api-1.0.jar
* pmml-model-1.1.7.jar
* pmml-schema-1.1.7.jar
This PR also upgrades `jaxb` and `activation`. We need to make sure this is a compatible change. Or we can use an older version of `jpmml`.
I will send you a PR that updates the code style. Then I will make another pass on the implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96950208
@mengxr please review, it should work as expected now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r29271370
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.pmml.export
+
+import scala.{Array => SArray}
+
+import org.dmg.pmml._
+
+import org.apache.spark.mllib.regression.GeneralizedLinearModel
+
+/**
+ * PMML Model Export for GeneralizedLinearModel class with binary ClassificationModel
+ */
+private[mllib] class BinaryClassificationPMMLModelExport(
+ model : GeneralizedLinearModel,
+ description : String,
+ normalizationMethod : RegressionNormalizationMethodType,
+ threshold: Double)
+ extends PMMLModelExport {
+
+ populateBinaryClassificationPMML()
+
+ /**
+ * Export the input LogisticRegressionModel or SVMModel to PMML format.
+ */
+ private def populateBinaryClassificationPMML(): Unit = {
+ pmml.getHeader.setDescription(description)
+
+ if (model.weights.size > 0) {
+ val fields = new SArray[FieldName](model.weights.size)
+ val dataDictionary = new DataDictionary
+ val miningSchema = new MiningSchema
+ val regressionTableYES = new RegressionTable(model.intercept).withTargetCategory("1")
+ var interceptNO = threshold
+ if (RegressionNormalizationMethodType.LOGIT == normalizationMethod) {
+ if (threshold <= 0)
+ interceptNO = -1000
+ else if (threshold >= 1)
+ interceptNO = 1000
--- End diff --
`Double.MaxValue`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97146364
[Test build #31147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31147/consoleFull) for PR 3062 at commit [`30165c4`](https://github.com/apache/spark/commit/30165c4caaea77b5b91ce2bef0ca94cb0454f527).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19771068
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
+
+ var miningSchema = new MiningSchema()
+
+ var comparisonMeasure = new ComparisonMeasure()
+ .withKind(Kind.DISTANCE)
+ .withMeasure(new SquaredEuclidean()
+ );
+
+ var clusteringModel = new ClusteringModel(miningSchema, comparisonMeasure,
+ MiningFunctionType.CLUSTERING, ModelClass.CENTER_BASED, model.clusterCenters.length)
+ .withModelName("k-means");
+
+ for ( i <- 0 to (clusterCenter.size - 1)) {
+ fields(i) = FieldName.create("field_" + i)
--- End diff --
Yeah, when trying to populate the data dictionary I realized the model didn't have the notion of the schema. At first I was thinking that the list of field names could be provided when exporting to PMML but really this should be in the model itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61797485
[Test build #22932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22932/consoleFull) for PR 3062 at commit [`ae8b993`](https://github.com/apache/spark/commit/ae8b993735e77d8c8c17ddf838379c5667c1d6ab).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61793044
Yes looking nice. @mengxr what do you think about `private[mllib]`? keep it internal for now? if it's exposed, I think some of the traits may have to change for Java interoperability.
My big question now is just whether this helps much if you can't identify the fields!
You could ask users to pass in the names of the fields, although that feels a bit ugly.
It feels like something that could get schema info from the upcoming Dataset API changes. Maybe it can wait until then, or if it's committed, stay internal-only for the moment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97341602
@mengxr updated based on your latest comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97601085
[Test build #31339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31339/consoleFull) for PR 3062 at commit [`852aac6`](https://github.com/apache/spark/commit/852aac6149d4d0c932ff34e959d148419b369983).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64949598
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23947/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94303934
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30564/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96544425
For binary logistic regression, using the same principle (intercept as threshold), doing some maths, we could set:
`intercept = -ln(1/threshold - 1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-75429812
@jkbradley I could add that too in addition or alternative to the local file.
In terms of implementation, I was thinking that the quick way of doing it would be:
def toPMML(sc: SparkContext, path: String) : Unit = {
val pmml = toPMML() //this export the model to a String
sc.parallelize(Array(pmml)).saveAsTextFile(path)
}
Is this OK or would you suggest a different approach?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-66890756
[Test build #24437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24437/consoleFull) for PR 3062 at commit [`8fe12bb`](https://github.com/apache/spark/commit/8fe12bb58358c0d08ede99ed9520e386aedf5110).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95830975
@selvinsource I tested KMeans.toPMML, which looks good. Linear regressors look good too, but for linear classifiers `LogisticRegressionModel` and `SVMModel`:
1. We should use `functionName="classification"` (as documented in http://www.dmg.org/v4-2-1/Regression.html). The following is an example logistic regression model in PMML (copied from the url above):
~~~xml
<PMML xmlns="http://www.dmg.org/PMML-4_2" version="4.2">
<Header copyright="DMG.org"/>
<DataDictionary numberOfFields="5">
<DataField name="age" optype="continuous" dataType="double"/>
<DataField name="work" optype="continuous" dataType="double"/>
<DataField name="sex" optype="categorical" dataType="string">
<Value value="0"/>
<Value value="1"/>
</DataField>
<DataField name="minority" optype="categorical" dataType="integer">
<Value value="0"/>
<Value value="1"/>
</DataField>
<DataField name="jobcat" optype="categorical" dataType="string">
<Value value="clerical"/>
<Value value="professional"/>
<Value value="trainee"/>
<Value value="skilled"/>
</DataField>
</DataDictionary>
<RegressionModel modelName="Sample for logistic regression" functionName="classification" algorithmName="logisticRegression" normalizationMethod="softmax" targetFieldName="jobcat">
<MiningSchema>
<MiningField name="age"/>
<MiningField name="work"/>
<MiningField name="sex"/>
<MiningField name="minority"/>
<MiningField name="jobcat" usageType="target"/>
</MiningSchema>
<RegressionTable intercept="46.418" targetCategory="clerical">
<NumericPredictor name="age" exponent="1" coefficient="-0.132"/>
<NumericPredictor name="work" exponent="1" coefficient="7.867E-02"/>
<CategoricalPredictor name="sex" value="0" coefficient="-20.525"/>
<CategoricalPredictor name="sex" value="1" coefficient="0.5"/>
<CategoricalPredictor name="minority" value="0" coefficient="-19.054"/>
<CategoricalPredictor name="minority" value="1" coefficient="0"/>
</RegressionTable>
<RegressionTable intercept="51.169" targetCategory="professional">
<NumericPredictor name="age" exponent="1" coefficient="-0.302"/>
<NumericPredictor name="work" exponent="1" coefficient="0.155"/>
<CategoricalPredictor name="sex" value="0" coefficient="-21.389"/>
<CategoricalPredictor name="sex" value="1" coefficient="0.1"/>
<CategoricalPredictor name="minority" value="0" coefficient="-18.443"/>
<CategoricalPredictor name="minority" value="1" coefficient="0"/>
</RegressionTable>
<RegressionTable intercept="25.478" targetCategory="trainee">
<NumericPredictor name="age" exponent="1" coefficient="-0.154"/>
<NumericPredictor name="work" exponent="1" coefficient="0.266"/>
<CategoricalPredictor name="sex" value="0" coefficient="-2.639"/>
<CategoricalPredictor name="sex" value="1" coefficient="0.8"/>
<CategoricalPredictor name="minority" value="0" coefficient="-19.821"/>
<CategoricalPredictor name="minority" value="1" coefficient="0.2"/>
</RegressionTable>
<RegressionTable intercept="0.0" targetCategory="skilled"/>
</RegressionModel>
</PMML>
~~~
2. `threshold` is not encoded in the PMML, which should be there.
3. We don't support multinomial logistic regression model. So we should check the number of classes and throw an exception if the model is multiclass.
I think we should fix 1&2 and throw exception for 3 in this PR. The multi-class support can come later. Since the feature freeze deadline is close, do you have time to update this PR before next Wed (leaving some buffer for review)? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96549372
Here my thinking.
When normalizationMethod = logit, the predicted value is computed as
`pj = 1 / ( 1 + exp( -yj ) )`
When we set `intercept = 0` for `targetCategory=0` => `yj = 0` => `pj = 1 / ( 1 + exp( 0 ) ) = 0.5`, which is in fact the default threshold for binary logistic classification.
Say we want `pj = threshold` (whereas threshold could be any value between 0 and 1, different to the default 0.5), we need to compute yj in terms of pj (which is given)
`yj = -ln(1/pj-1)` where yj is the intercept and pj is the threshold, hence `intercept = -ln(1/threshold - 1)`.
Need to be tested, meantime, what do @vruusmann @mengxr think?
A pratical example, `model.threshold = 0.6` => `intercept = -ln(1/threshold - 1) = 0.41` (calculated from pmml exporter) => `pj = 1 / ( 1 + exp( -0.41 ) ) = 0.6` (computed by JPMML back to the original threshold).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97620088
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by tvmanikandan <gi...@git.apache.org>.
Github user tvmanikandan commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-138593339
when PMML support for multi class expected to be released ? I am eagerly waiting PMML multiclass support for Logistic Regression , stochastic gradient descent (SGD) and SVM.
Please let me know.
-Mani
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-95067402
[Test build #30733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30733/consoleFull) for PR 3062 at commit [`25dce33`](https://github.com/apache/spark/commit/25dce336065e39da8492c6d8379a21ab3733cd45).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-66890757
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24437/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-96406891
Sorry, I only verified SVM and thought logistic regression was implemented the same way. For SVM, could we try `normalizationMethod = none` and set the threshold as the intercept of the second regression table?
~~~xml
<RegressionModel functionName="classification" normalizationMethod="none" ...
<RegressionTable targetCategory="YES" ...
<RegressionTable targetCategory="NO" intercept="0.05"
...
~~~
Maybe we can do the same for binary logistic regression. I don't know whether this is the correct way to do it by looking at the PMML specification. Need help from @vruusmann
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94759397
@mengxr @vruusmann
I am using 1.1.7, therefore compatible with Java 6. Should I update it to the latest Java 6 supported version 1.1.15? Any benefit in doing so?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94959269
@selvinsource Let's upgrade to 1.1.15 and remove FastInfoset and istack-commons-runtime from the build as @vruusmann recommended. Then we need to check the transitive dependencies again.
Is JAXB backward compatible?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61445290
This won't tell you much of anything you don't already know but here's a bit of code also making a K-means model with JPMML: https://github.com/OryxProject/oryx/blob/master/oryx-ml-mllib/src/main/java/com/cloudera/oryx/ml/mllib/kmeans/KMeansUpdate.java#L121
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64870641
[Test build #23935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23935/consoleFull) for PR 3062 at commit [`78515ec`](https://github.com/apache/spark/commit/78515ec9d8403d555b996163ca32409d91cb30f7).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-72565380
@mengxr I like keeping the export together in one package. Since I imagine there will likely only ever be 1 or 2 model export formats (really, I imagine 1) I think using a trait is OK. That is, I don't feel too bad about treating PMML fairly specially.
The `PMML` abstraction from JPMML is pretty solid and could be the return type of such a `toPMML` API method, although I hesitate to bake in a third-party library class to the public API. I suppose the return type could just be `String`, containing an XML document, or at best a generic object representing the XML DOM?
Yes, definitely mark things private / developer API until this stabilizes.
Writing PMML serializations is a moderate PITA, but using the JPMML library internally makes it not so bad. I have some serializations already written I could chuck into the pot once the framework is in place (decision forest, k-means clustering model)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97601009
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19720393
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/KMeansPMMLModelExport.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.clustering.KMeansModel
+import org.dmg.pmml.DataDictionary
+import org.dmg.pmml.FieldName
+import org.dmg.pmml.DataField
+import org.dmg.pmml.OpType
+import org.dmg.pmml.DataType
+import org.dmg.pmml.MiningSchema
+import org.dmg.pmml.MiningField
+import org.dmg.pmml.FieldUsageType
+import org.dmg.pmml.ComparisonMeasure
+import org.dmg.pmml.ComparisonMeasure.Kind
+import org.dmg.pmml.SquaredEuclidean
+import org.dmg.pmml.ClusteringModel
+import org.dmg.pmml.MiningFunctionType
+import org.dmg.pmml.ClusteringModel.ModelClass
+import org.dmg.pmml.ClusteringField
+import org.dmg.pmml.CompareFunctionType
+import org.dmg.pmml.Cluster
+import org.dmg.pmml.Array.Type
+
+/**
+ * PMML Model Export for KMeansModel class
+ */
+class KMeansPMMLModelExport(model : KMeansModel) extends PMMLModelExport{
+
+ /**
+ * Export the input KMeansModel model to PMML format
+ */
+ populateKMeansPMML(model);
+
+ private def populateKMeansPMML(model : KMeansModel): Unit = {
+
+ pmml.getHeader().setDescription("k-means clustering");
+
+ if(model.clusterCenters.length > 0){
+
+ val clusterCenter = model.clusterCenters(0)
+
+ var fields = new Array[FieldName](clusterCenter.size)
+
+ var dataDictionary = new DataDictionary()
+
+ var miningSchema = new MiningSchema()
+
+ var comparisonMeasure = new ComparisonMeasure()
+ .withKind(Kind.DISTANCE)
+ .withMeasure(new SquaredEuclidean()
+ );
+
+ var clusteringModel = new ClusteringModel(miningSchema, comparisonMeasure,
+ MiningFunctionType.CLUSTERING, ModelClass.CENTER_BASED, model.clusterCenters.length)
+ .withModelName("k-means");
+
+ for ( i <- 0 to (clusterCenter.size - 1)) {
--- End diff --
use `until clusterCenter.size` in Scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-76614926
[Test build #28149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28149/consoleFull) for PR 3062 at commit [`b25bbf7`](https://github.com/apache/spark/commit/b25bbf7ae7eb92541f59f5cfe7495280d60fca43).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64949597
[Test build #23947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23947/consoleFull) for PR 3062 at commit [`19adf29`](https://github.com/apache/spark/commit/19adf29d0bf2c5ffbdd7ba257296475463dd5eae).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97620494
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-76622800
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28149/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61812768
@srowen
For the fields I have the same opinion, the exporter shouldn't ask the user to provide the fields name, the model should be aware of that and I am glad this is upcoming.
It would be great if it is committed at least internally and, I assume, it can be used from the spark-shell. There are scenarios whereas it useful, for exampe: load the exported pmml file in JPMML Evaluator and use it to do the prediction without the need of tying the app to Spark.
I attached to Jira the updated version of the PDF and the PMML xml generated by the ModelExporter object. For reference I attached my test class that proved JPMML Evaluator gives the same result of KMeansModel.predict.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19793638
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/ModelExport.scala ---
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export
+
+import java.io.OutputStream
+import java.io.FileOutputStream
+import java.io.File
+
+trait ModelExport {
--- End diff --
Yes, a utility method makes some sense, just not as part of the type hierarchy perhaps. You can expose a utility object that reads/writes PMML objects separately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94968132
@mengxr @vruusmann, I tried the upgrade to 1.1.15 and it requires some code change, I will look into that.
Once we have the new list of dependencies I will review NOTICE and LICENSE as suggested by @srowen.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94751334
[Test build #30661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30661/consoleFull) for PR 3062 at commit [`a0a55f7`](https://github.com/apache/spark/commit/a0a55f7add73b0fbbb27b285e9316cb23c716aaf).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `FastInfoset-1.2.12.jar`
* `istack-commons-runtime-2.16.jar`
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `jsr173_api-1.0.jar`
* `pmml-model-1.1.7.jar`
* `pmml-schema-1.1.7.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-73427139
[Test build #27048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27048/consoleFull) for PR 3062 at commit [`7a949d0`](https://github.com/apache/spark/commit/7a949d0b3f5b397f4df4fcd70f3658cc66e8f04e).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (val clusterCenters: Array[Vector]) extends Serializable with PMMLExportable `
* `trait PMMLExportable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r29271374
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.pmml.export
+
+import scala.{Array => SArray}
+
+import org.dmg.pmml._
+
+import org.apache.spark.mllib.regression.GeneralizedLinearModel
+
+/**
+ * PMML Model Export for GeneralizedLinearModel class with binary ClassificationModel
+ */
+private[mllib] class BinaryClassificationPMMLModelExport(
+ model : GeneralizedLinearModel,
+ description : String,
+ normalizationMethod : RegressionNormalizationMethodType,
+ threshold: Double)
+ extends PMMLModelExport {
+
+ populateBinaryClassificationPMML()
+
+ /**
+ * Export the input LogisticRegressionModel or SVMModel to PMML format.
+ */
+ private def populateBinaryClassificationPMML(): Unit = {
+ pmml.getHeader.setDescription(description)
+
+ if (model.weights.size > 0) {
+ val fields = new SArray[FieldName](model.weights.size)
+ val dataDictionary = new DataDictionary
+ val miningSchema = new MiningSchema
+ val regressionTableYES = new RegressionTable(model.intercept).withTargetCategory("1")
+ var interceptNO = threshold
+ if (RegressionNormalizationMethodType.LOGIT == normalizationMethod) {
+ if (threshold <= 0)
+ interceptNO = -1000
+ else if (threshold >= 1)
+ interceptNO = 1000
+ else
+ interceptNO = -math.log(1/threshold -1)
--- End diff --
space around `/` and after `-`: `math.log(1 / threshold - 1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94976619
@selvinsource Sounds good. Please keep the code style consistent:)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19769244
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import org.apache.spark.mllib.export.ModelExport
+import java.io.OutputStream
+import org.jpmml.model.JAXBUtil
+import org.dmg.pmml.PMML
+import javax.xml.transform.stream.StreamResult
+import scala.beans.BeanProperty
+import org.dmg.pmml.Application
+import org.dmg.pmml.Timestamp
+import org.dmg.pmml.Header
+import java.text.SimpleDateFormat
+import java.util.Date
+
+trait PMMLModelExport extends ModelExport{
+
+ /**
+ * Holder of the exported model in PMML format
+ */
+ @BeanProperty
+ var pmml: PMML = new PMML();
+
+ setHeader(pmml);
+
+ private def setHeader(pmml : PMML): Unit = {
+ var version = getClass().getPackage().getImplementationVersion()
+ var app = new Application().withName("Apache Spark MLlib").withVersion(version)
+ var timestamp = new Timestamp()
+ .withContent(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date()))
+ var header = new Header()
+ .withCopyright("www.dmg.org")
--- End diff --
I wasn't sure what to put here.
Rattle put DMG
http://www.dmg.org/pmml_examples/rattle_pmml_examples/IrisKMeans.xml
while KNIME put KNIME
http://www.dmg.org/pmml_examples/KNIME_PMML_4.1_Examples/single_iris_kmeans.xml
Should I change it to "Apache Spark"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97620554
[Test build #31357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31357/consoleFull) for PR 3062 at commit [`852aac6`](https://github.com/apache/spark/commit/852aac6149d4d0c932ff34e959d148419b369983).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61792414
@srowen I added a wrapper called ModelExporter and changed everything else to private[mllib], therefore this should be the only object exposed in the API.
I also separated, as you suggested, the model export hierarchy from the serialization (which is now in the helper ModelExporter).
The usage is now even simpler
`ModelExporter.toPMML(kmeansModel,"/tmp/kmeans.xml")`
or any stream supported by the StreamResult
`ModelExporter.toPMML(kmeansModel,new StreamResult(System.out))`
I also committed some other changes based on your other comments above (added the pmml version, ordered the imports, change the copyright to Apache Spark MLlib).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61407019
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97614088
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31339/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61788148
[Test build #22931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22931/consoleFull) for PR 3062 at commit [`6357b98`](https://github.com/apache/spark/commit/6357b9890e6f4245a2e8a7d0f90dd6d80495e60a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61788707
[Test build #22932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22932/consoleFull) for PR 3062 at commit [`ae8b993`](https://github.com/apache/spark/commit/ae8b993735e77d8c8c17ddf838379c5667c1d6ab).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61416984
[Test build #22763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22763/consoleFull) for PR 3062 at commit [`aba5ee1`](https://github.com/apache/spark/commit/aba5ee1d75b012a17a1db077bdd88c0970310cae).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94303930
[Test build #30564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30564/consoleFull) for PR 3062 at commit [`1676e15`](https://github.com/apache/spark/commit/1676e15ff5bec346466a095a593efdd324f6b515).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel (`
* `trait PMMLExportable `
* This patch **adds the following new dependencies:**
* `FastInfoset-1.2.12.jar`
* `istack-commons-runtime-2.16.jar`
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `jsr173_api-1.0.jar`
* `pmml-model-1.1.7.jar`
* `pmml-schema-1.1.7.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94653600
[Test build #30646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30646/consoleFull) for PR 3062 at commit [`e2313df`](https://github.com/apache/spark/commit/e2313df33560bfd1e8377ff33be8a67d5ad0922a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61813085
[Test build #22938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22938/consoleFull) for PR 3062 at commit [`e29dfb9`](https://github.com/apache/spark/commit/e29dfb99ffecb3cd78e5f2a1357b2244bc9c9737).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61554631
Hi Sean,
first of all thanks for your time in reviewing this.
If you don't foresee any other export format I agree that the ModelExport is not really needed and can be removed together with the ModelExportType. At that point I would move the ModelExportFactory to the pmml package and call it PMMLModelExportFactory.
Let me know how to proceed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-94704278
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30646/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61419910
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22763/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61797491
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22932/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-72566799
@srowen I expect us to have 2 model formats (PMML, plus a Spark-specific one supporting distributed models). +1 for returning a String or some type which lets us hide the JPMML dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-64864182
[Test build #23935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23935/consoleFull) for PR 3062 at commit [`78515ec`](https://github.com/apache/spark/commit/78515ec9d8403d555b996163ca32409d91cb30f7).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-97555744
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/3062#issuecomment-61416819
@srowen Could you help review this? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1406] Mllib pmml model export
Posted by selvinsource <gi...@git.apache.org>.
Github user selvinsource commented on a diff in the pull request:
https://github.com/apache/spark/pull/3062#discussion_r19870913
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/export/pmml/PMMLModelExport.scala ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.export.pmml
+
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import scala.beans.BeanProperty
+
+import org.dmg.pmml.Application
+import org.dmg.pmml.Header
+import org.dmg.pmml.PMML
+import org.dmg.pmml.Timestamp
+
+import org.apache.spark.mllib.export.ModelExport
+
+private[mllib] trait PMMLModelExport extends ModelExport{
+
+ /**
+ * Holder of the exported model in PMML format
+ */
+ @BeanProperty
+ val pmml: PMML = new PMML();
+
+ setHeader(pmml);
+
+ private def setHeader(pmml : PMML): Unit = {
+ val version = getClass().getPackage().getImplementationVersion()
+ val app = new Application().withName("Apache Spark MLlib").withVersion(version)
+ val timestamp = new Timestamp()
+ .withContent(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date()))
+ val header = new Header()
+ .withCopyright("Apache Spark MLlib")
+ .withApplication(app)
+ .withTimestamp(timestamp);
+ pmml.setHeader(header);
+ pmml.setVersion("4.2")
--- End diff --
I checked and by default JPMML uses 4.2, so I removed the hardcoded 4.2 I added before.
https://github.com/jpmml/jpmml-model/blob/master/pmml-schema/src/main/java/org/jpmml/schema/Version.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org