You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangmiao1981 <gi...@git.apache.org> on 2016/03/08 20:03:58 UTC
[GitHub] spark pull request: SPARK-13034
GitHub user wangmiao1981 opened a pull request:
https://github.com/apache/spark/pull/11582
SPARK-13034
I added Import and Export for Logisticregression and Naive Bayes
Test ./python/run-tests --python-executables=python2.7 --modules=pyspark-ml
Result:
Running PySpark tests. Output is in /Users/mwang/bugfix/spark_bug_fix/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-ml']
Finished test(python2.7): pyspark.ml.feature (26s)
Finished test(python2.7): pyspark.ml.recommendation (27s)
Finished test(python2.7): pyspark.ml.classification (27s)
Finished test(python2.7): pyspark.ml.clustering (27s)
Finished test(python2.7): pyspark.ml.evaluation (12s)
Finished test(python2.7): pyspark.ml.tuning (17s)
Finished test(python2.7): pyspark.ml.regression (21s)
Finished test(python2.7): pyspark.ml.tests (21s)
Tests passed in 48 seconds
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangmiao1981/spark ml_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11582.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11582
----
commit bb04f6fb11f605daff9f7971f70b477d7f490af9
Author: Miao Wang <wa...@users.noreply.github.com>
Date: 2015-10-09T22:28:33Z
Merge pull request #1 from apache/master
Update
commit df949c50b2d0a771690ecbc23c5a2691ea155233
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-10-16T17:19:40Z
Merge remote-tracking branch 'upstream/master'
Get new spark
commit 7fc01ac3047062d6d3a0db5d1ba63045f8fec3fa
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-10-21T18:07:19Z
Merge https://github.com/apache/spark
commit 7fd0842050383a57148e3953bd38e204d1d28c5d
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-10-21T18:09:00Z
fix the merge issue
commit f8260b3fe1497d0ced2c8c8390cf3ca1344fed12
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-10-22T21:18:14Z
Merge remote-tracking branch 'upstream/master'
commit f797fb6a262878cbe7cdcf821a4f05444a9cc617
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-10-29T18:07:09Z
Merge remote-tracking branch 'upstream/master'
commit 319a8801ad946929403f22a3814376f9e4631668
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-11-10T19:17:59Z
Merge remote-tracking branch 'upstream/master'
commit 0cde5199113873681937ce797e25b43eae7c0cca
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-11-19T19:49:34Z
Merge remote-tracking branch 'upstream/master'
commit 3db1e259a2621d47d23c21ef4a8a6e8d2aaafbdc
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-11-30T18:16:17Z
Merge remote-tracking branch 'upstream/master'
commit 2b2e455aea12601dc68e5bbe7d88ec87dfe74b7a
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-12-07T22:05:19Z
Merge remote-tracking branch 'upstream/master'
commit 233bc48e6c5b855c42fd5598234f0b2f5f92eea3
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-12-14T19:47:18Z
Merge remote-tracking branch 'upstream/master'
commit 5dc0ab2e84fbd39021e65ca8d0cffbf71251243c
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-12-16T19:00:59Z
Merge remote-tracking branch 'upstream/master'
commit f94698f5feef14bff933681d53848cc42f56a4b6
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2015-12-18T18:42:13Z
Merge remote-tracking branch 'upstream/master'
commit 2efb49afb7f4d154d4511f604e643f34335c1c78
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-01-04T19:29:15Z
Merge remote-tracking branch 'upstream/master'
commit 8bba263ad3e750bc37186cd8665a4f5ff0028a03
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-01-06T19:42:44Z
Merge remote-tracking branch 'upstream/master'
commit 67089d17c247dc3b6612d55bbaa01c3ac6ac5cd5
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-01-06T22:01:44Z
Merge remote-tracking branch 'upstream/master'
commit e88ee81a41e6d50af130785936c0017c3c401e36
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-01-19T18:22:53Z
Merge remote-tracking branch 'upstream/master'
commit 4359e34dc69b2b8e2d5714fe1c7894a70ad5cb21
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-01-26T20:05:25Z
Merge remote-tracking branch 'upstream/master'
commit ac90a3c4c1233e3c03ad527a9b6ccde3e2636350
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-01-29T18:06:52Z
Merge remote-tracking branch 'upstream/master'
commit 585c63324efc060b9abac7ba8a668840a756b4d2
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-05T18:38:01Z
Merge remote-tracking branch 'upstream/master'
commit 7adabdaec46fa39fc61ff3c966a06a36a1350fbc
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-08T23:20:01Z
Merge remote-tracking branch 'upstream/master'
commit 388fa5ec8de340fdcb480c60773b9d2d1121e39c
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-09T18:57:20Z
Merge remote-tracking branch 'upstream/master'
commit 248b513ee8b31ad45be030eac4fa105f483ea54c
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-09T19:44:33Z
Merge remote-tracking branch 'upstream/master'
commit ff193f2d6f7dae6b56e1950839675171d6e46c57
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-09T22:06:38Z
Merge remote-tracking branch 'upstream/master'
commit 2a9af156020f200e4dfbb384d5032060863f468a
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-25T00:41:39Z
Merge remote-tracking branch 'upstream/master'
commit 98f92b22ac9fcc8e337176eceb51b49a8babe1b3
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-25T23:52:17Z
Merge remote-tracking branch 'upstream/master'
commit 5cec79a95054bf5f32c1f6a52f4e31fcff58ac26
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-02-27T01:05:29Z
Merge remote-tracking branch 'upstream/master'
commit a79614985cd29fc614b8f338f3ec888151bface1
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-03-04T00:48:26Z
commit import/export version 1
commit 07084c1da0301349684552b48553855931bdffa0
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-03-06T22:58:16Z
check in the logisticregression fix
commit ca406a420cadeace09cc195c4170aa48fb380bd8
Author: wm624@hotmail.com <wm...@hotmail.com>
Date: 2016-03-06T23:03:51Z
check in the logisticregression fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-193919805
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-194531149
@GayathriMurali @wangmiao1981 It will be great to add persistence for LogisticRegression and NaiveBayes. The other algorithms will need to wait since they do not yet have Scala support for persistence. Could you please reconcile your 2 PRs? It would be great to close one and have both of you collaborate on 1 PR. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-196668602
Also, to "fix the title," you'll need to put square brackets around the JIRA: "[SPARK-13034]" and probably remove the colon ":"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-196668151
Thanks both for collaborating!
@wangmiao1981 If you want to get rid of the commits merged from master, then you can do ```git rebase upstream/master```. Feel free to do that now to clean up the commits. In general, though, try not to rebase a PR unless there are merge conflicts because it makes the PR history harder to understand.
Can you please revert the changes to the files in ```bin/```? It looks like you changed the permissions and committed that change.
I'll take a look at the other parts now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-194014892
@srowen I added the title in the pull request. Sorry for causing the confusion here. I only made changes in one python file. All other changes are merged from master to my private branch. I don't know why they are included in the PULL request. I will update it and clean it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-198064456
close this one as it has been merged with 11707.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-193960757
@wangmiao1981 can you fix the title? and maybe squash your commits once upfront here. There are 32 already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-196671658
It looks like [https://github.com/apache/spark/pull/11707] is closer to being merged now, so let's go with it instead. @wangmiao1981 Could you please close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by wangmiao1981 <gi...@git.apache.org>.
Github user wangmiao1981 closed the pull request at:
https://github.com/apache/spark/pull/11582
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by GayathriMurali <gi...@git.apache.org>.
Github user GayathriMurali commented on the pull request:
https://github.com/apache/spark/pull/11582#issuecomment-195602622
@wangmiao1981 I closed my PR. I have made the changes to add persistence to only LogisticRegression and Naive Bayes. Would you like to review? We could then send one PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-13034[ML]:PySpark ml.classification supp...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11582#discussion_r56116371
--- Diff: sql/hive/pom.xml ---
@@ -38,6 +38,11 @@
<dependencies>
<!-- Added for Hive Parquet SerDe -->
<dependency>
+ <groupId>org.scala-lang</groupId>
--- End diff --
Please revert the change to this file; it should not be needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org