You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by ScrapCodes <gi...@git.apache.org> on 2014/03/14 13:25:16 UTC
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
GitHub user ScrapCodes opened a pull request:
https://github.com/apache/spark/pull/140
SPARK-1246, added min max API to Double RDDs in java and scala APIs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ScrapCodes/spark-1 SPARK-1246/min-max
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/140.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #140
----
commit 0b20bc758a41bc483be4e258f7031cc02969c206
Author: Prashant Sharma <pr...@imaginea.com>
Date: 2014-03-14T12:24:18Z
SPARK-1246, added min max API to Double RDDs in java and scala APIs.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/140#issuecomment-37642654
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/140#discussion_r10632880
--- Diff: core/src/main/scala/org/apache/spark/rdd/DoubleRDDFunctions.scala ---
@@ -86,14 +92,9 @@ class DoubleRDDFunctions(self: RDD[Double]) extends Logging with Serializable {
* If the elements in RDD do not vary (max == min) always returns a single bucket.
*/
def histogram(bucketCount: Int): Pair[Array[Double], Array[Long]] = {
- // Compute the minimum and the maxium
- val (max: Double, min: Double) = self.mapPartitions { items =>
- Iterator(items.foldRight(Double.NegativeInfinity,
- Double.PositiveInfinity)((e: Double, x: Pair[Double, Double]) =>
- (x._1.max(e), x._2.min(e))))
- }.reduce { (maxmin1, maxmin2) =>
- (maxmin1._1.max(maxmin2._1), maxmin1._2.min(maxmin2._2))
- }
+ // Compute the minimum and the maximum from stats once
+ val _stats = stats()
+ val (max: Double, min: Double) = (_stats.max, _stats.min)
--- End diff --
Okay, will change that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes closed the pull request at:
https://github.com/apache/spark/pull/140
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/140#discussion_r10618038
--- Diff: core/src/main/scala/org/apache/spark/rdd/DoubleRDDFunctions.scala ---
@@ -86,14 +92,9 @@ class DoubleRDDFunctions(self: RDD[Double]) extends Logging with Serializable {
* If the elements in RDD do not vary (max == min) always returns a single bucket.
*/
def histogram(bucketCount: Int): Pair[Array[Double], Array[Long]] = {
- // Compute the minimum and the maxium
- val (max: Double, min: Double) = self.mapPartitions { items =>
- Iterator(items.foldRight(Double.NegativeInfinity,
- Double.PositiveInfinity)((e: Double, x: Pair[Double, Double]) =>
- (x._1.max(e), x._2.min(e))))
- }.reduce { (maxmin1, maxmin2) =>
- (maxmin1._1.max(maxmin2._1), maxmin1._2.min(maxmin2._2))
- }
+ // Compute the minimum and the maximum from stats once
+ val _stats = stats()
+ val (max: Double, min: Double) = (_stats.max, _stats.min)
--- End diff --
Minor, but I'd prefer not to unnecessarily bind a val in a scope where it doesn't need to be used:
```scala
// Compute the minimum and the maximum from stats once
val (max, min) = { val sc = stats(); (sc.max, sc.min) }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/140#issuecomment-37642655
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/140#discussion_r10632860
--- Diff: project/build.properties ---
@@ -14,4 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-sbt.version=0.13.1
+sbt.version=0.13.2-M1
--- End diff --
It was accidental, (sorry about that.). I use this version of sbt locally since its really fast with incremental builds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/140#issuecomment-37647359
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13180/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/140#issuecomment-37647358
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1246, added min max API to Double RDDs i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/140#discussion_r10616091
--- Diff: project/build.properties ---
@@ -14,4 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-sbt.version=0.13.1
+sbt.version=0.13.2-M1
--- End diff --
Is there a reason for this? I'd rather not depend on a milestone release of sbt because it's more likely to have bugs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---