You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by blazy2k9 <gi...@git.apache.org> on 2016/04/26 15:00:49 UTC
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
GitHub user blazy2k9 opened a pull request:
https://github.com/apache/spark/pull/12701
Unintentional white spaces in kryo classes configuration parameters
## What changes were proposed in this pull request?
Pruned off white spaces present in the user provided comma separated list of classes for **spark.kryo.classesToRegister** and **spark.kryo.registrator**.
## How was this patch tested?
Manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/blazy2k9/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12701.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12701
----
commit 0c5790f6b3c26900491b94c3b40d2bded0f3be75
Author: Victor Chima <bl...@gmail.com>
Date: 2016-04-13T08:40:43Z
Handle case of unintentional white space(s) in the comma separated list
of classes passed to spark.kryo.registrator and
spark.kryo.classesToRegister parameters.
commit 5395c4e4564ef47d7f5121d28b448c2e85fca229
Author: Victor Chima <bl...@gmail.com>
Date: 2016-04-26T12:20:25Z
Revert "Handle case of unintentional white space(s) in the comma separated list"
This reverts commit 0c5790f6b3c26900491b94c3b40d2bded0f3be75.
commit a5cad3122375cdb849fd5d09cb0d32713660208f
Author: Victor Chima <bl...@gmail.com>
Date: 2016-04-26T12:26:17Z
Handle case of unintentional white space(s) in the comma separated list of classes passed to spark.kryo.registrator and spark.kryo.classesToRegister parameters.
commit 482b1fb38e60cdddfaed62f3b14cb55aa1770952
Author: Victor Chima <bl...@gmail.com>
Date: 2016-04-26T12:51:40Z
Merge branch 'master' of github.com:apache/spark
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-214768953
Please also read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-215037982
**[Test build #57102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57102/consoleFull)** for PR 12701 at commit [`e1714d1`](https://github.com/apache/spark/commit/e1714d182dc5651c8cba9e8cd0e1dcb7a977e319).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-215004010
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61098280
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -191,7 +191,8 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
*/
def registerKryoClasses(classes: Array[Class[_]]): SparkConf = {
val allClassNames = new LinkedHashSet[String]()
- allClassNames ++= get("spark.kryo.classesToRegister", "").split(',').filter(!_.isEmpty)
+ allClassNames ++= get("spark.kryo.classesToRegister", "").replaceAll("\\s+", "").split(',')
--- End diff --
Easier -- `.trim` after splitting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61098206
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -22,10 +22,10 @@ import java.util.concurrent.ConcurrentHashMap
import scala.collection.JavaConverters._
import scala.collection.mutable.LinkedHashSet
-import org.apache.avro.{Schema, SchemaNormalization}
+import org.apache.avro.{ Schema, SchemaNormalization }
--- End diff --
These spaces should not be added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-215038162
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by blazy2k9 <gi...@git.apache.org>.
Github user blazy2k9 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61218165
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -191,7 +191,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
*/
def registerKryoClasses(classes: Array[Class[_]]): SparkConf = {
val allClassNames = new LinkedHashSet[String]()
- allClassNames ++= get("spark.kryo.classesToRegister", "").split(',').filter(!_.isEmpty)
+ allClassNames ++= get("spark.kryo.classesToRegister", "").split(',').trim.filter(!_.isEmpty)
--- End diff --
Yes, I just fixed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-215038165
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57102/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-215128460
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-214768869
Most of this change is unrelated to your purpose, just whitespace changes. Some of it is OK but some is not; here I'd just make changes related to whitespace around names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61186521
--- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -71,10 +71,10 @@ class KryoSerializer(conf: SparkConf)
private val referenceTracking = conf.getBoolean("spark.kryo.referenceTracking", true)
private val registrationRequired = conf.getBoolean("spark.kryo.registrationRequired", false)
private val userRegistrators = conf.get("spark.kryo.registrator", "")
- .split(',')
+ .split(',').trim
--- End diff --
Ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-214733660
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12701
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61186507
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -191,7 +191,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
*/
def registerKryoClasses(classes: Array[Class[_]]): SparkConf = {
val allClassNames = new LinkedHashSet[String]()
- allClassNames ++= get("spark.kryo.classesToRegister", "").split(',').filter(!_.isEmpty)
+ allClassNames ++= get("spark.kryo.classesToRegister", "").split(',').trim.filter(!_.isEmpty)
--- End diff --
Isn't `trim` being applied to `Array[String]`?
I think this should be `.split(',').map(_.trim)`
```scala
scala> "A, B, C".split(",").trim
<console>:8: error: value trim is not a member of Array[String]
"A, B, C".split(",").trim
^
```
```scala
scala> "A, B, C".split(",").map(_.trim)
res2: Array[String] = Array(A, B, C)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by blazy2k9 <gi...@git.apache.org>.
Github user blazy2k9 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61103479
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -191,7 +191,8 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
*/
def registerKryoClasses(classes: Array[Class[_]]): SparkConf = {
val allClassNames = new LinkedHashSet[String]()
- allClassNames ++= get("spark.kryo.classesToRegister", "").split(',').filter(!_.isEmpty)
+ allClassNames ++= get("spark.kryo.classesToRegister", "").replaceAll("\\s+", "").split(',')
--- End diff --
I will remove the white spaces added and do a .trim instead of a .replace
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12701#discussion_r61186530
--- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -71,10 +71,10 @@ class KryoSerializer(conf: SparkConf)
private val referenceTracking = conf.getBoolean("spark.kryo.referenceTracking", true)
private val registrationRequired = conf.getBoolean("spark.kryo.registrationRequired", false)
private val userRegistrators = conf.get("spark.kryo.registrator", "")
- .split(',')
+ .split(',').trim
.filter(!_.isEmpty)
private val classesToRegister = conf.get("spark.kryo.classesToRegister", "")
- .split(',')
+ .split(',').trim
--- End diff --
Ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Unintentional white spaces in kryo classes con...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12701#issuecomment-215004605
**[Test build #57102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57102/consoleFull)** for PR 12701 at commit [`e1714d1`](https://github.com/apache/spark/commit/e1714d182dc5651c8cba9e8cd0e1dcb7a977e319).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org