You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/24 00:32:40 UTC

[GitHub] [spark] zhengruifeng opened a new pull request #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

zhengruifeng opened a new pull request #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990
 
 
   ### What changes were proposed in this pull request?
   compute count and quantile on one pass
   
   ### Why are the changes needed?
   to avoid extra pass
   
   
   ### Does this PR introduce any user-facing change?
   No
   
   
   ### How was this patch tested?
   existing testsuites
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568619397
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568619401
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115665/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568614034
 
 
   **[Test build #115665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115665/testReport)** for PR 26990 at commit [`586d947`](https://github.com/apache/spark/commit/586d94760217bdb83fb00d27b03d087c333d4311).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-569253078
 
 
   @srowen 
   
   > approxQuantile basically do this already under the hood?
   
   Yes, approxQuantile also use QuantileSummaries like this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568727845
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-569487655
 
 
   Merged to master, thanks @srowen @huaxingao for reviewing!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568709043
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20517/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568709038
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] huaxingao commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-569145154
 
 
   The change looks good to me. Seems that approxQuantile does this under the hood. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568727661
 
 
   **[Test build #115723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115723/testReport)** for PR 26990 at commit [`586d947`](https://github.com/apache/spark/commit/586d94760217bdb83fb00d27b03d087c333d4311).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568614214
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568614034
 
 
   **[Test build #115665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115665/testReport)** for PR 26990 at commit [`586d947`](https://github.com/apache/spark/commit/586d94760217bdb83fb00d27b03d087c333d4311).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568614219
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20462/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568619311
 
 
   **[Test build #115665 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115665/testReport)** for PR 26990 at commit [`586d947`](https://github.com/apache/spark/commit/586d94760217bdb83fb00d27b03d087c333d4311).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568708681
 
 
   **[Test build #115723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115723/testReport)** for PR 26990 at commit [`586d947`](https://github.com/apache/spark/commit/586d94760217bdb83fb00d27b03d087c333d4311).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568619397
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
srowen commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-569303198
 
 
   I'm OK with it. It sounds like an optimization without much if any downside.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568839015
 
 
   friendly ping @huaxingao @srowen 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568614219
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20462/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568619401
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115665/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568727849
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115723/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568614214
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568727845
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a change in pull request #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#discussion_r361033143
 
 

 ##########
 File path: mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala
 ##########
 @@ -112,9 +113,7 @@ private[ml] abstract class LSHModel[T <: LSHModel[T]]
       numNearestNeighbors: Int,
       singleProbe: Boolean,
       distCol: String): Dataset[_] = {
-    val count = dataset.count()
-    require(numNearestNeighbors > 0 && numNearestNeighbors <= count, "The number of" +
-      " nearest neighbors cannot be less than 1 or greater than the number of elements in dataset")
+    require(numNearestNeighbors > 0, "The number of nearest neighbors cannot be less than 1")
 
 Review comment:
   As discussed in https://github.com/apache/spark/pull/26415,
   
   "@param numNearestNeighbors The maximum number of nearest neighbors."
   It implies it could return fewer items.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568709038
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng closed pull request #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568709043
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20517/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568708681
 
 
   **[Test build #115723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115723/testReport)** for PR 26990 at commit [`586d947`](https://github.com/apache/spark/commit/586d94760217bdb83fb00d27b03d087c333d4311).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568727849
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115723/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #26990: [SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization
URL: https://github.com/apache/spark/pull/26990#issuecomment-568707273
 
 
   retest this please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org