You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/22 09:40:14 UTC
[GitHub] [spark] wetneb opened a new pull request #28293: SPARK-31518: Expose filterByRange in JavaPairRDD.
wetneb opened a new pull request #28293:
URL: https://github.com/apache/spark/pull/28293
### What changes were proposed in this pull request?
This exposes the `filterByRange` method from `OrderedRDDFunctions` in the Java API (as a method of JavaPairRDD).
This is the only method of `OrderedRDDFunctions` which is not exposed in the Java API so far.
### Why are the changes needed?
This improves the consistency between the Scala and Java APIs. Calling the Scala method manually from a Java context is cumbersome as it requires passing many ClassTags.
### Does this PR introduce any user-facing change?
Yes, a new method in the Java API.
### How was this patch tested?
With unit tests. The implementation of the Scala method is already tested independently and it was not touched in this PR.
Suggesting @srowen as a reviewer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618116914
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413152556
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
Just for curiosity, of the two since marks, Spark's scala @Since annotation and java @since tag, how to choose them? IMHO, @since tag seems better here to let the version show up in the generated Java API documentation. @dongjoon-hyun @srowen thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617935007
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121630/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618075869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413157180
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
For the generated Java API doc, you should put `@since` into the comment.
```
/**
* Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
* If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
* performed efficiently by only scanning the partitions that might containt matching elements.
* Otherwise, a standard `filter` is applied to all partitions.
* @since 3.1.0
*/
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617934144
**[Test build #121630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121630/testReport)** for PR 28293 at commit [`b74661d`](https://github.com/apache/spark/commit/b74661d133c567221e59b0760ebd30981d1c5502).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618149931
@wetneb . You are added to Apache Spark contributor group and SPARK-31518 is assigned to you. Thank you again, @wetneb !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413157180
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
For the generated Java API doc, you should put `@since` into the comment.
```
/**
* Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
* If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
* performed efficiently by only scanning the partitions that might containt matching elements.
* Otherwise, a standard `filter` is applied to all partitions.
*
* @since 3.1.0
*/
```
For example,
- https://dist.apache.org/repos/dist/dev/spark/v3.0.0-rc1-docs/_site/api/java/org/apache/spark/sql/DataFrameNaFunctions.html
- https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala#L33
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618075518
**[Test build #121640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121640/testReport)** for PR 28293 at commit [`62dd41c`](https://github.com/apache/spark/commit/62dd41ce5456a437f4c283d3f657c22759809809).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617834732
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617673090
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617834116
**[Test build #121630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121630/testReport)** for PR 28293 at commit [`b74661d`](https://github.com/apache/spark/commit/b74661d133c567221e59b0760ebd30981d1c5502).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413123935
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
Thank you for your first contribution, @wetneb ! +1 for @srowen 's answers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618116914
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413157180
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
For the generated Java API doc, you should put `@since` into the comment.
```
/**
* Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
* If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
* performed efficiently by only scanning the partitions that might containt matching elements.
* Otherwise, a standard `filter` is applied to all partitions.
*
* @since 3.1.0
*/
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617934996
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413248774
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -247,6 +247,34 @@ public int getPartition(Object key) {
Arrays.asList(new Tuple2<>(1, 3), new Tuple2<>(3, 8), new Tuple2<>(3, 8)));
}
+ @Test
+ public void filterByRange() {
+ List<Tuple2<Integer, Integer>> pairs = new ArrayList<>();
+ pairs.add(new Tuple2<>(0, 5));
+ pairs.add(new Tuple2<>(1, 8));
+ pairs.add(new Tuple2<>(2, 6));
+ pairs.add(new Tuple2<>(3, 8));
+ pairs.add(new Tuple2<>(4, 8));
+
+ JavaPairRDD<Integer, Integer> rdd = sc.parallelizePairs(pairs).sortByKey();
+
+ // Default comparator
+ JavaPairRDD<Integer, Integer> filteredRDD = rdd.filterByRange(3, 11);
+ List<Tuple2<Integer, Integer>> filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 2);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(3, 8));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(4, 8));
+
+ // Custom comparator
+ filteredRDD = rdd.filterByRange(Collections.reverseOrder(), 3, -2);
+ filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 4);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(0, 5));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(1, 8));
+ assertEquals(filteredPairs.get(2), new Tuple2<>(2, 6));
+ assertEquals(filteredPairs.get(3), new Tuple2<>(3, 9));
Review comment:
BTW, @wetneb . Could you switch the parameter of `assertEquals`? The first parameter should be `expected` and the second parameter is `actual`.
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -247,6 +247,34 @@ public int getPartition(Object key) {
Arrays.asList(new Tuple2<>(1, 3), new Tuple2<>(3, 8), new Tuple2<>(3, 8)));
}
+ @Test
+ public void filterByRange() {
+ List<Tuple2<Integer, Integer>> pairs = new ArrayList<>();
+ pairs.add(new Tuple2<>(0, 5));
+ pairs.add(new Tuple2<>(1, 8));
+ pairs.add(new Tuple2<>(2, 6));
+ pairs.add(new Tuple2<>(3, 8));
+ pairs.add(new Tuple2<>(4, 8));
+
+ JavaPairRDD<Integer, Integer> rdd = sc.parallelizePairs(pairs).sortByKey();
+
+ // Default comparator
+ JavaPairRDD<Integer, Integer> filteredRDD = rdd.filterByRange(3, 11);
+ List<Tuple2<Integer, Integer>> filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 2);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(3, 8));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(4, 8));
+
+ // Custom comparator
+ filteredRDD = rdd.filterByRange(Collections.reverseOrder(), 3, -2);
+ filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 4);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(0, 5));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(1, 8));
+ assertEquals(filteredPairs.get(2), new Tuple2<>(2, 6));
+ assertEquals(filteredPairs.get(3), new Tuple2<>(3, 9));
Review comment:
BTW, @wetneb . Could you switch the parameters of `assertEquals`? The first parameter should be `expected` and the second parameter is `actual`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617830874
Jenkins test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wetneb commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
wetneb commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413322313
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -247,6 +247,34 @@ public int getPartition(Object key) {
Arrays.asList(new Tuple2<>(1, 3), new Tuple2<>(3, 8), new Tuple2<>(3, 8)));
}
+ @Test
+ public void filterByRange() {
+ List<Tuple2<Integer, Integer>> pairs = new ArrayList<>();
+ pairs.add(new Tuple2<>(0, 5));
+ pairs.add(new Tuple2<>(1, 8));
+ pairs.add(new Tuple2<>(2, 6));
+ pairs.add(new Tuple2<>(3, 8));
+ pairs.add(new Tuple2<>(4, 8));
+
+ JavaPairRDD<Integer, Integer> rdd = sc.parallelizePairs(pairs).sortByKey();
+
+ // Default comparator
+ JavaPairRDD<Integer, Integer> filteredRDD = rdd.filterByRange(3, 11);
+ List<Tuple2<Integer, Integer>> filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 2);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(3, 8));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(4, 8));
+
+ // Custom comparator
+ filteredRDD = rdd.filterByRange(Collections.reverseOrder(), 3, -2);
+ filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 4);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(0, 5));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(1, 8));
+ assertEquals(filteredPairs.get(2), new Tuple2<>(2, 6));
+ assertEquals(filteredPairs.get(3), new Tuple2<>(3, 9));
Review comment:
Oops, sorry about this! I thought I had managed to run the test, but I obviously need to improve my set up. Many apologies!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617934996
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wetneb commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
wetneb commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413059789
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
Sure. I am also wondering whether it makes sense to backport this in 2.4 by the way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618074266
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wetneb commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
wetneb commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413167481
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
Thanks, I have added that too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413062793
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
No, it's not a bug fix per se. I wouldn't put it in 3.0 even necessarily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617673670
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413248046
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -247,6 +247,34 @@ public int getPartition(Object key) {
Arrays.asList(new Tuple2<>(1, 3), new Tuple2<>(3, 8), new Tuple2<>(3, 8)));
}
+ @Test
+ public void filterByRange() {
+ List<Tuple2<Integer, Integer>> pairs = new ArrayList<>();
+ pairs.add(new Tuple2<>(0, 5));
+ pairs.add(new Tuple2<>(1, 8));
+ pairs.add(new Tuple2<>(2, 6));
+ pairs.add(new Tuple2<>(3, 8));
+ pairs.add(new Tuple2<>(4, 8));
+
+ JavaPairRDD<Integer, Integer> rdd = sc.parallelizePairs(pairs).sortByKey();
+
+ // Default comparator
+ JavaPairRDD<Integer, Integer> filteredRDD = rdd.filterByRange(3, 11);
+ List<Tuple2<Integer, Integer>> filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 2);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(3, 8));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(4, 8));
+
+ // Custom comparator
+ filteredRDD = rdd.filterByRange(Collections.reverseOrder(), 3, -2);
+ filteredPairs = filteredRDD.collect();
+ assertEquals(filteredPairs.size(), 4);
+ assertEquals(filteredPairs.get(0), new Tuple2<>(0, 5));
+ assertEquals(filteredPairs.get(1), new Tuple2<>(1, 8));
+ assertEquals(filteredPairs.get(2), new Tuple2<>(2, 6));
+ assertEquals(filteredPairs.get(3), new Tuple2<>(3, 9));
Review comment:
```
sbt.ForkMain$ForkError: java.lang.AssertionError: expected:<(3,8)> but was:<(3,9)>
at org.junit.Assert.fail(Assert.java:88)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617834732
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617834116
**[Test build #121630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121630/testReport)** for PR 28293 at commit [`b74661d`](https://github.com/apache/spark/commit/b74661d133c567221e59b0760ebd30981d1c5502).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617673090
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618116329
**[Test build #121640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121640/testReport)** for PR 28293 at commit [`62dd41c`](https://github.com/apache/spark/commit/62dd41ce5456a437f4c283d3f657c22759809809).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618075869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413152556
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
Just for curiosity, of the two since marks, Spark's scala `@Since` annotation and java `@since` tag, how to choose them? IMHO, `@since` tag seems better here to let the version show up in the generated Java API documentation. @dongjoon-hyun @srowen thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-617673670
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28293:
URL: https://github.com/apache/spark/pull/28293#issuecomment-618075518
**[Test build #121640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121640/testReport)** for PR 28293 at commit [`62dd41c`](https://github.com/apache/spark/commit/62dd41ce5456a437f4c283d3f657c22759809809).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413057803
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
You'll want the `@Since("3.1.0")` annotations on these methods
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413157180
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
For the generated Java API doc, you should put `@since` into the comment.
```
/**
* Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
* If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
* performed efficiently by only scanning the partitions that might containt matching elements.
* Otherwise, a standard `filter` is applied to all partitions.
*
* @since 3.1.0
*/
```
- https://dist.apache.org/repos/dist/dev/spark/v3.0.0-rc1-docs/_site/api/java/org/apache/spark/sql/DataFrameNaFunctions.html
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
For the generated Java API doc, you should put `@since` into the comment.
```
/**
* Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
* If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
* performed efficiently by only scanning the partitions that might containt matching elements.
* Otherwise, a standard `filter` is applied to all partitions.
*
* @since 3.1.0
*/
```
For example,
- https://dist.apache.org/repos/dist/dev/spark/v3.0.0-rc1-docs/_site/api/java/org/apache/spark/sql/DataFrameNaFunctions.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28293: [SPARK-31518][CORE] Expose filterByRange in JavaPairRDD
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28293:
URL: https://github.com/apache/spark/pull/28293#discussion_r413157180
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
##########
@@ -937,6 +937,28 @@ class JavaPairRDD[K, V](val rdd: RDD[(K, V)])
fromRDD(new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey(ascending, numPartitions))
}
+ /**
+ * Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
+ * If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
+ * performed efficiently by only scanning the partitions that might containt matching elements.
+ * Otherwise, a standard `filter` is applied to all partitions.
+ */
+ def filterByRange(lower: K, upper: K): JavaPairRDD[K, V] = {
Review comment:
For the generated Java API doc, you should put `@since` into the comment.
```
/**
* Return a RDD containing only the elements in the inclusive range `lower` to `upper`.
* If the RDD has been partitioned using a `RangePartitioner`, then this operation can be
* performed efficiently by only scanning the partitions that might containt matching elements.
* Otherwise, a standard `filter` is applied to all partitions.
* @since 3.1.0
*/
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org