You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zsxwing <gi...@git.apache.org> on 2014/04/25 09:52:31 UTC
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/549
SPARK-1628: Add missing hashCode methods in Partitioner subclasses
JIRA: https://issues.apache.org/jira/browse/SPARK-1628
Added `hashCode` in HashPartitioner, RangePartitioner, PythonPartitioner and PageRankUtils.CustomPartitioner.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-1628
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/549.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #549
----
commit d5c32282ee64f96a1fdbf45609565bad474ac7a1
Author: zsxwing <zs...@gmail.com>
Date: 2014-04-25T07:10:37Z
SPARK-1628: Add missing hashCode methods in Partitioner subclasses
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-42122245
Is there any further suggestion about this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45430500
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11994957
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -155,4 +157,17 @@ class RangePartitioner[K : Ordering : ClassTag, V](
case _ =>
false
}
+
+
+ override def hashCode(): Int = {
+ val prime = 31
+ var result = 1
+ var i = 0
+ while (i < rangeBounds.length) {
--- End diff --
Darn, `java.util.Arrays.hashCode(Object[])` doesn't match any scala array? `java.util.Arrays.hashCode(Array(1,2,3))` works fine but that's not quite the situation here. Oh well. Maybe wrap as a `List` and use its `hashCode`? may not be worth it to save the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11994568
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -155,4 +157,17 @@ class RangePartitioner[K : Ordering : ClassTag, V](
case _ =>
false
}
+
+
+ override def hashCode(): Int = {
+ val prime = 31
+ var result = 1
+ var i = 0
+ while (i < rangeBounds.length) {
--- End diff --
There is not a generics `Arrays.hashCode`. `Arrays.hashCode(rangeBounds)` cannot be compiled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11989683
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonPartitioner.scala ---
@@ -50,4 +50,12 @@ private[spark] class PythonPartitioner(
case _ =>
false
}
+
+ override def hashCode: Int = {
--- End diff --
This is probably simpler as `31 * numPartitions + pyPartitionFunctionId.hashCode` instead of even these 5 lines? same idea above too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45431187
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15541/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11994594
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonPartitioner.scala ---
@@ -50,4 +50,12 @@ private[spark] class PythonPartitioner(
case _ =>
false
}
+
+ override def hashCode: Int = {
--- End diff --
Sure. I will update it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45430517
This looks good to me pending test passes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11995767
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -155,4 +157,17 @@ class RangePartitioner[K : Ordering : ClassTag, V](
case _ =>
false
}
+
+
+ override def hashCode(): Int = {
+ val prime = 31
+ var result = 1
+ var i = 0
+ while (i < rangeBounds.length) {
--- End diff --
Maybe Scala compiler can not determine that using which one here: Arrays.hashCode(Object[]), Arrays.hashCode(int[]), or Arrays.hashCode(double[])...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45448899
Thanks. I'm merging this in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11989658
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -155,4 +157,17 @@ class RangePartitioner[K : Ordering : ClassTag, V](
case _ =>
false
}
+
+
+ override def hashCode(): Int = {
+ val prime = 31
+ var result = 1
+ var i = 0
+ while (i < rangeBounds.length) {
--- End diff --
Could this be simplified a lot with Arrays.hashCode()?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/549#discussion_r11986353
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -119,7 +121,7 @@ class RangePartitioner[K : Ordering : ClassTag, V](
}
}
- def numPartitions = partitions
+ def numPartitions = rangeBounds.length + 1
--- End diff --
If using `numPartitions = partitions`, there is a chance that `p1 == p2 && p1.numPartitions != p2.numPartitions` is true. For example, if `rdd.sample` is empty, `p1 = new RangePartitioner[...](10, rdd, true)`, and `p2 = new RangePartitioner[...](1, rdd, true)`.
That's confusing. So I changed `partitions` to `rangeBounds.length + 1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45430600
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/549
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45431186
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-41367441
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1628: Add missing hashCode methods in Pa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/549#issuecomment-45430604
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---