You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tmnd1991 <gi...@git.apache.org> on 2016/06/04 08:42:22 UTC
[GitHub] spark pull request #13509: SPARK-15740
GitHub user tmnd1991 opened a pull request:
https://github.com/apache/spark/pull/13509
SPARK-15740
## What changes were proposed in this pull request?
"test big model load / save" in Word2VecSuite, lately resulted into OOM.
Therefore we decided to make the partitioning adaptive (not based on spark default "spark.kryoserializer.buffer.max" conf) and then testing it using a small buffer size in order to trigger partitioning without allocating too much memory for the test.
## How was this patch tested?
It was tested running the following unit test:
org.apache.spark.mllib.feature.Word2VecSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tmnd1991/spark SPARK-15740
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13509.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13509
----
commit d5c766851b5b6286db3db70941f8c8953acbb1de
Author: tmnd1991 <an...@studio.unibo.it>
Date: 2016-06-04T08:05:58Z
Make partitioning adaptive, set low memory size for kryo buffer to trigger partitioning
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/13509
I don't think you can access the default confs in this case. The class KryoSerializer seems to store those privately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3164/consoleFull)** for PR 13509 at commit [`909b6e1`](https://github.com/apache/spark/commit/909b6e16cbca29a2eaaecfd4151c5ac0af546cae).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3164 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3164/consoleFull)** for PR 13509 at commit [`909b6e1`](https://github.com/apache/spark/commit/909b6e16cbca29a2eaaecfd4151c5ac0af546cae).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3113 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3113/consoleFull)** for PR 13509 at commit [`dfcd850`](https://github.com/apache/spark/commit/dfcd8505f92d6d704e5eac60ca1828aa0a222008).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3111/consoleFull)** for PR 13509 at commit [`dfcd850`](https://github.com/apache/spark/commit/dfcd8505f92d6d704e5eac60ca1828aa0a222008).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3113 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3113/consoleFull)** for PR 13509 at commit [`dfcd850`](https://github.com/apache/spark/commit/dfcd8505f92d6d704e5eac60ca1828aa0a222008).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model lo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13509
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model lo...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13509#discussion_r67939360
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -91,20 +91,39 @@ class Word2VecSuite extends SparkFunSuite with MLlibTestSparkContext {
}
- ignore("big model load / save") {
- // create a model bigger than 32MB since 9000 * 1000 * 4 > 2^25
- val word2VecMap = Map((0 to 9000).map(i => s"$i" -> Array.fill(1000)(0.1f)): _*)
+ test("big model load / save") {
+ // backupping old values
+ val oldBufferConfValue = spark.conf.get("spark.kryoserializer.buffer.max", "64m")
+ val oldBufferMaxConfValue = spark.conf.get("spark.kryoserializer.buffer", "64k")
+
+ // setting test values to trigger partitioning
+ spark.conf.set("spark.kryoserializer.buffer", "50b")
+ spark.conf.set("spark.kryoserializer.buffer.max", "50b")
+
+ // create a model bigger than 50 Bytes
+ val word2VecMap = Map((0 to 10).map(i => s"$i" -> Array.fill(10)(0.1f)): _*)
val model = new Word2VecModel(word2VecMap)
+ // est. size of this model, given the formula:
+ // (floatSize * vectorSize + 15) * numWords
+ // (4 * 10 + 15) * 10 = 550
+ // therefore it should generate 12 partitions
val tempDir = Utils.createTempDir()
val path = tempDir.toURI.toString
try {
model.save(sc, path)
val sameModel = Word2VecModel.load(sc, path)
assert(sameModel.getVectors.mapValues(_.toSeq) === model.getVectors.mapValues(_.toSeq))
- } finally {
+ }
+ catch {
--- End diff --
style: put on previous line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by tmnd1991 <gi...@git.apache.org>.
Github user tmnd1991 commented on the issue:
https://github.com/apache/spark/pull/13509
Can anyone verify this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by tmnd1991 <gi...@git.apache.org>.
Github user tmnd1991 commented on the issue:
https://github.com/apache/spark/pull/13509
I noticed a scala style error, wait till new commit before triggering a jenkins build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/13509
Merging with master and branch-2.0
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model lo...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13509#discussion_r67939365
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -91,20 +91,39 @@ class Word2VecSuite extends SparkFunSuite with MLlibTestSparkContext {
}
- ignore("big model load / save") {
- // create a model bigger than 32MB since 9000 * 1000 * 4 > 2^25
- val word2VecMap = Map((0 to 9000).map(i => s"$i" -> Array.fill(1000)(0.1f)): _*)
+ test("big model load / save") {
+ // backupping old values
+ val oldBufferConfValue = spark.conf.get("spark.kryoserializer.buffer.max", "64m")
+ val oldBufferMaxConfValue = spark.conf.get("spark.kryoserializer.buffer", "64k")
+
+ // setting test values to trigger partitioning
+ spark.conf.set("spark.kryoserializer.buffer", "50b")
+ spark.conf.set("spark.kryoserializer.buffer.max", "50b")
+
+ // create a model bigger than 50 Bytes
+ val word2VecMap = Map((0 to 10).map(i => s"$i" -> Array.fill(10)(0.1f)): _*)
val model = new Word2VecModel(word2VecMap)
+ // est. size of this model, given the formula:
+ // (floatSize * vectorSize + 15) * numWords
+ // (4 * 10 + 15) * 10 = 550
+ // therefore it should generate 12 partitions
val tempDir = Utils.createTempDir()
val path = tempDir.toURI.toString
try {
model.save(sc, path)
val sameModel = Word2VecModel.load(sc, path)
assert(sameModel.getVectors.mapValues(_.toSeq) === model.getVectors.mapValues(_.toSeq))
- } finally {
+ }
+ catch {
+ case t: Throwable => fail("exception thrown persisting a model " +
+ "that spans over 12 partitions", t)
--- End diff --
"12 partitions" --> "multiple partitions" (The exact number isn't important.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: SPARK-15740
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13509
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model lo...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13509#discussion_r67939355
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -91,20 +91,39 @@ class Word2VecSuite extends SparkFunSuite with MLlibTestSparkContext {
}
- ignore("big model load / save") {
- // create a model bigger than 32MB since 9000 * 1000 * 4 > 2^25
- val word2VecMap = Map((0 to 9000).map(i => s"$i" -> Array.fill(1000)(0.1f)): _*)
+ test("big model load / save") {
+ // backupping old values
+ val oldBufferConfValue = spark.conf.get("spark.kryoserializer.buffer.max", "64m")
+ val oldBufferMaxConfValue = spark.conf.get("spark.kryoserializer.buffer", "64k")
+
+ // setting test values to trigger partitioning
+ spark.conf.set("spark.kryoserializer.buffer", "50b")
+ spark.conf.set("spark.kryoserializer.buffer.max", "50b")
+
+ // create a model bigger than 50 Bytes
+ val word2VecMap = Map((0 to 10).map(i => s"$i" -> Array.fill(10)(0.1f)): _*)
val model = new Word2VecModel(word2VecMap)
+ // est. size of this model, given the formula:
+ // (floatSize * vectorSize + 15) * numWords
+ // (4 * 10 + 15) * 10 = 550
+ // therefore it should generate 12 partitions
--- End diff --
"12 partitions" --> "multiple partitions" (The exact number isn't important.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3112 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3112/consoleFull)** for PR 13509 at commit [`dfcd850`](https://github.com/apache/spark/commit/dfcd8505f92d6d704e5eac60ca1828aa0a222008).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3111 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3111/consoleFull)** for PR 13509 at commit [`dfcd850`](https://github.com/apache/spark/commit/dfcd8505f92d6d704e5eac60ca1828aa0a222008).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model lo...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/13509#discussion_r67939366
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -91,20 +91,39 @@ class Word2VecSuite extends SparkFunSuite with MLlibTestSparkContext {
}
- ignore("big model load / save") {
- // create a model bigger than 32MB since 9000 * 1000 * 4 > 2^25
- val word2VecMap = Map((0 to 9000).map(i => s"$i" -> Array.fill(1000)(0.1f)): _*)
+ test("big model load / save") {
+ // backupping old values
+ val oldBufferConfValue = spark.conf.get("spark.kryoserializer.buffer.max", "64m")
+ val oldBufferMaxConfValue = spark.conf.get("spark.kryoserializer.buffer", "64k")
+
+ // setting test values to trigger partitioning
+ spark.conf.set("spark.kryoserializer.buffer", "50b")
+ spark.conf.set("spark.kryoserializer.buffer.max", "50b")
+
+ // create a model bigger than 50 Bytes
+ val word2VecMap = Map((0 to 10).map(i => s"$i" -> Array.fill(10)(0.1f)): _*)
val model = new Word2VecModel(word2VecMap)
+ // est. size of this model, given the formula:
+ // (floatSize * vectorSize + 15) * numWords
+ // (4 * 10 + 15) * 10 = 550
+ // therefore it should generate 12 partitions
val tempDir = Utils.createTempDir()
val path = tempDir.toURI.toString
try {
model.save(sc, path)
val sameModel = Word2VecModel.load(sc, path)
assert(sameModel.getVectors.mapValues(_.toSeq) === model.getVectors.mapValues(_.toSeq))
- } finally {
+ }
+ catch {
+ case t: Throwable => fail("exception thrown persisting a model " +
+ "that spans over 12 partitions", t)
+ }
+ finally {
--- End diff --
style: put on previous line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3166 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3166/consoleFull)** for PR 13509 at commit [`909b6e1`](https://github.com/apache/spark/commit/909b6e16cbca29a2eaaecfd4151c5ac0af546cae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3166 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3166/consoleFull)** for PR 13509 at commit [`909b6e1`](https://github.com/apache/spark/commit/909b6e16cbca29a2eaaecfd4151c5ac0af546cae).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: SPARK-15740
Posted by tmnd1991 <gi...@git.apache.org>.
Github user tmnd1991 commented on the issue:
https://github.com/apache/spark/pull/13509
(Fix the title please) https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/13509
I verified locally that the test creates a model file with multiple partitions, so LGTM
I'll merge once tests run again.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by tmnd1991 <gi...@git.apache.org>.
Github user tmnd1991 commented on the issue:
https://github.com/apache/spark/pull/13509
I corrected the style errors you pointed out. If you say I cannot retrieve default values, I will leave the 64m hard coded that way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/13509
I triggered multiple test runs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by tmnd1991 <gi...@git.apache.org>.
Github user tmnd1991 commented on the issue:
https://github.com/apache/spark/pull/13509
The only thing I don't like is that "64m" hard coded, but I couldn't find where default spark confs are stored!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13509: [SPARK-15740] [MLLIB] Word2VecSuite "big model load / sa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13509
**[Test build #3112 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3112/consoleFull)** for PR 13509 at commit [`dfcd850`](https://github.com/apache/spark/commit/dfcd8505f92d6d704e5eac60ca1828aa0a222008).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org