You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/08 05:44:55 UTC
[GitHub] [spark] LuciferYang opened a new pull request #31517: [SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
LuciferYang opened a new pull request #31517:
URL: https://github.com/apache/spark/pull/31517
### What changes were proposed in this pull request?
Caffeine is a high performance, near optimal caching library based on Java 8, it is used in a similar way to guava cache, but with better performance. The main purpose of this pr is Use Caffeine instead of Guava Cache.
### Why are the changes needed?
Use better local cache lib.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888779189
@holdenk 6c74fc6 merge with master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887177451
**[Test build #141677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141677/testReport)** for PR 31517 at commit [`9cd9c35`](https://github.com/apache/spark/commit/9cd9c35872094b0f60f5175dc85494c45cde10d8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887180729
**[Test build #141679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141679/testReport)** for PR 31517 at commit [`7b360a7`](https://github.com/apache/spark/commit/7b360a7d577ea379db3487d451b0c7a744d1dc02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806333716
**[Test build #136493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136493/testReport)** for PR 31517 at commit [`045d3dc`](https://github.com/apache/spark/commit/045d3dc50eb6c719ae3ff8f071b09b618c24aab2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809682146
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41243/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810080553
**[Test build #136708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136708/testReport)** for PR 31517 at commit [`3ba2574`](https://github.com/apache/spark/commit/3ba25749cf0ce67b3758ccc4db9d407ffa39309d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603741576
##########
File path: pom.xml
##########
@@ -484,6 +484,16 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ <version>2.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>guava</artifactId>
+ <version>2.9.0</version>
+ </dependency>
Review comment:
because Java 11 or above is required for 3.x and it is no longer compatible with Java 8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805809994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-803730936
**[Test build #136324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136324/testReport)** for PR 31517 at commit [`adc6d92`](https://github.com/apache/spark/commit/adc6d92f1e2d9f02a31198a21d4c306592c23715).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-803787983
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774997465
**[Test build #135011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135011/testReport)** for PR 31517 at commit [`4b49b84`](https://github.com/apache/spark/commit/4b49b84e0c038d286ca09039e774815f4aea7296).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805825918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805531674
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136432/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-804742873
**[Test build #136395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136395/testReport)** for PR 31517 at commit [`8842fa3`](https://github.com/apache/spark/commit/8842fa36d731d0558768a7ae486114baedb8ab19).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810216422
**[Test build #136709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136709/testReport)** for PR 31517 at commit [`8995d72`](https://github.com/apache/spark/commit/8995d72ecd64ec6407769dbd8545c046898b816b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-878005489
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140893/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887220339
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141679/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-815459515
**[Test build #137047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137047/testReport)** for PR 31517 at commit [`f61b041`](https://github.com/apache/spark/commit/f61b0410491f6cdc75bdf51dfc13857a6cd5b65a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r600205859
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -223,7 +224,10 @@ private[history] class ApplicationCache(
}
}
- def invalidate(key: CacheKey): Unit = appCache.invalidate(key)
+ def invalidate(key: CacheKey): Unit = appCache.asMap().computeIfPresent(key, (key, value) => {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887312795
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872302730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805646534
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136450/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871936226
> Take your time, @LuciferYang . This should target Apache Spark 3.3 because Apache Spark 3.2 branch cut is July 1st.
>
> There are some busy cases today, we'll try to fix the ignored case tomorrow
>
> BTW, cc @viirya and @sunchao
@dongjoon-hyun Thank you for your information. I will update the commit message for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805857975
**[Test build #136467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136467/testReport)** for PR 31517 at commit [`32d9ce0`](https://github.com/apache/spark/commit/32d9ce0ca259fe4fa61424aa3288a62232bc4164).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888838992
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46319/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876914660
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45348/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-881191734
**[Test build #141137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141137/testReport)** for PR 31517 at commit [`db9cc56`](https://github.com/apache/spark/commit/db9cc566813507c91983092db876bb86ebf67600).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806024661
**[Test build #136467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136467/testReport)** for PR 31517 at commit [`32d9ce0`](https://github.com/apache/spark/commit/32d9ce0ca259fe4fa61424aa3288a62232bc4164).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810190465
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136708/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-877980413
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603560075
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -62,21 +64,27 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
- override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]): Unit = {
+ override def onRemoval(key: CacheKey, value: CacheEntry,
+ cause: RemovalCause): Unit = {
metrics.evictionCount.inc()
- val key = rm.getKey
- logDebug(s"Evicting entry ${key}")
- operations.detachSparkUI(key.appId, key.attemptId, rm.getValue().loadedUI.ui)
+ logDebug(s"Evicting entry $key")
+ operations.detachSparkUI(key.appId, key.attemptId, value.loadedUI.ui)
}
}
private val appCache: LoadingCache[CacheKey, CacheEntry] = {
- CacheBuilder.newBuilder()
- .maximumSize(retainedApplications)
- .removalListener(removalListener)
- .build(appLoader)
+ val builder = Caffeine.newBuilder()
+ .maximumSize(retainedApplications)
+ .removalListener(removalListener)
+ // SPARK-34309: Use custom Executor to compatible with
+ // the data eviction behavior of Guava cache
+ .executor((command: Runnable) => command.run())
+ // Wrapping as CaffeinatedGuava to be compatible with
+ // the exception behavior of Guava cache
+ CaffeinatedGuava.build(builder, appLoader)
Review comment:
Is there a reason to preserve behavior of guava cache here ?
If there are better alternatives, we should consider them.
##########
File path: pom.xml
##########
@@ -484,6 +484,16 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ <version>2.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>guava</artifactId>
+ <version>2.9.0</version>
+ </dependency>
Review comment:
Any particular reason to use 2.x when 3.x is out ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806529210
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136504/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806316574
**[Test build #136491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136491/testReport)** for PR 31517 at commit [`cf30234`](https://github.com/apache/spark/commit/cf30234e0c4eb3dcd6d28101e834d067e0511b6d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871937797
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44993/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805531674
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136432/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806196333
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41058/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884860713
**[Test build #141491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141491/testReport)** for PR 31517 at commit [`0743e3b`](https://github.com/apache/spark/commit/0743e3b6e39f13213f5aa162fbd8bfe1f5c36cf5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876357463
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45302/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-877940872
**[Test build #140893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140893/testReport)** for PR 31517 at commit [`ee05314`](https://github.com/apache/spark/commit/ee053145c3a45a9f63fdb37c8c2b246498067376).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-897430503
> @LuciferYang Caffeine is definitely an amazing and popular project. But when we decide whether we should add a library , especially a popular library, to Spark, we also need to think about the impact to the downstream libraries and Spark applications. As Caffeine is pretty popular, many downstream libraries and Spark users probably are using it. If Spark adds it to its classpath, it will force them to use the same Caffeine version as Spark. If someone would like to use a different Caffeine version, the work would be non trivial since they need to shade their Caffeine version.
Can we solve this problem by `shade + relocation`, just like now Spark uses `Guava` and `Jetty` now. If we accept it, I can do this work.
> Do you have any examples that we can get Caffeine's benefits from hot path, so that we can use that to make the tradeoff?
For example, `FileStatusCache` on driver side of a resident process(like thrift-server). Of course, the improvement will not be very significant because Guava Cache is not bad.
Overall, I think both are OK. If need to revert the code, please tell me.I am not entangled and opposed to this. I can also be responsible for completing this work :)
I just want to mention it again. As I said [above](https://github.com/apache/spark/pull/31517#issuecomment-896504990), we are still using Guava 14.0.1, which is a 2013 version. Some fixed about Guava Cache and other Guava components are still hidden in Spark, and I found many JIRA want to upgrade it which seems to have not been completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876280059
**[Test build #140789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140789/testReport)** for PR 31517 at commit [`7506999`](https://github.com/apache/spark/commit/7506999636ec7fe73427bc3b1e2398c11a5dbf04).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887178881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46192/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805809994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872050821
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-804822321
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805919425
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41042/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873857198
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140635/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805811351
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41037/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-881281018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141137/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876939984
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45348/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876280059
**[Test build #140789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140789/testReport)** for PR 31517 at commit [`7506999`](https://github.com/apache/spark/commit/7506999636ec7fe73427bc3b1e2398c11a5dbf04).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r571795094
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -58,24 +58,26 @@ private[history] class ApplicationCache(
}
- private val removalListener = new RemovalListener[CacheKey, CacheEntry] {
+ private val cacheWriter = new CacheWriter[CacheKey, CacheEntry] {
Review comment:
`CacheWriter ` adopts sync remove behavior similar to guava and `RemovalListener ` always Asynchronous
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-804746044
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40975/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806316574
**[Test build #136491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136491/testReport)** for PR 31517 at commit [`cf30234`](https://github.com/apache/spark/commit/cf30234e0c4eb3dcd6d28101e834d067e0511b6d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r677081396
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
##########
@@ -136,11 +136,14 @@ private[spark] object BlockManagerId {
* The max cache size is hardcoded to 10000, since the size of a BlockManagerId
* object is about 48B, the total memory cost should be below 1MB which is feasible.
*/
- val blockManagerIdCache = CacheBuilder.newBuilder()
- .maximumSize(10000)
- .build(new CacheLoader[BlockManagerId, BlockManagerId]() {
- override def load(id: BlockManagerId) = id
- })
+ val blockManagerIdCache = {
Review comment:
I didn't realize that usage, my Scala level was too poor, thx ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887178755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141677/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887507448
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-881191734
**[Test build #141137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141137/testReport)** for PR 31517 at commit [`db9cc56`](https://github.com/apache/spark/commit/db9cc566813507c91983092db876bb86ebf67600).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809920635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876376157
**[Test build #140789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140789/testReport)** for PR 31517 at commit [`7506999`](https://github.com/apache/spark/commit/7506999636ec7fe73427bc3b1e2398c11a5dbf04).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedFieldPosition(position: ColumnPosition) extends FieldPosition `
* ` .doc(\"The custom cost evaluator class to be used for adaptive execution. If not being set,\" +`
* `case class RocksDBMetrics(`
* `case class RocksDBNativeHistogram(`
* `case class RocksDBFileManagerMetrics(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-889022884
**[Test build #141818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141818/testReport)** for PR 31517 at commit [`5a75b2c`](https://github.com/apache/spark/commit/5a75b2cf1c70d9317bec6a65bb4211027ff290d0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884003130
**[Test build #141380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141380/testReport)** for PR 31517 at commit [`c6b9dc6`](https://github.com/apache/spark/commit/c6b9dc69d20caf4732ee85872e8b103cebe8ba0f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ContinuousWriteRDD(var prev: RDD[InternalRow], writerFactory: StreamingDataWriterFactory,`
* `case class WriteToContinuousDataSource(write: StreamingWrite, query: LogicalPlan,`
* `case class WriteToContinuousDataSourceExec(write: StreamingWrite, query: SparkPlan,`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883288691
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888527104
**[Test build #141782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141782/testReport)** for PR 31517 at commit [`87def1a`](https://github.com/apache/spark/commit/87def1aac3b8030584b1696e20d2fe3232643153).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805816667
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136464/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805632644
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41026/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871972433
**[Test build #140493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140493/testReport)** for PR 31517 at commit [`34d31fd`](https://github.com/apache/spark/commit/34d31fdb2e274d3224c69d257051e03c00f0ca3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-815478471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137047/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ben-manes commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
ben-manes commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r600131955
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -223,7 +224,10 @@ private[history] class ApplicationCache(
}
}
- def invalidate(key: CacheKey): Unit = appCache.invalidate(key)
+ def invalidate(key: CacheKey): Unit = appCache.asMap().computeIfPresent(key, (key, value) => {
Review comment:
yes. I'd pass `RemovalCause.EXPLICIT`, just to avoid null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806568975
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41092/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-828345076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809658692
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41239/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810057888
**[Test build #136698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136698/testReport)** for PR 31517 at commit [`95d5c2e`](https://github.com/apache/spark/commit/95d5c2e36f4da54e5ad69e212ebdf9e20f32bf1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603577466
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -62,21 +64,27 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
- override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]): Unit = {
+ override def onRemoval(key: CacheKey, value: CacheEntry,
+ cause: RemovalCause): Unit = {
metrics.evictionCount.inc()
- val key = rm.getKey
- logDebug(s"Evicting entry ${key}")
- operations.detachSparkUI(key.appId, key.attemptId, rm.getValue().loadedUI.ui)
+ logDebug(s"Evicting entry $key")
+ operations.detachSparkUI(key.appId, key.attemptId, value.loadedUI.ui)
}
}
private val appCache: LoadingCache[CacheKey, CacheEntry] = {
- CacheBuilder.newBuilder()
- .maximumSize(retainedApplications)
- .removalListener(removalListener)
- .build(appLoader)
+ val builder = Caffeine.newBuilder()
+ .maximumSize(retainedApplications)
+ .removalListener(removalListener)
+ // SPARK-34309: Use custom Executor to compatible with
+ // the data eviction behavior of Guava cache
+ .executor((command: Runnable) => command.run())
+ // Wrapping as CaffeinatedGuava to be compatible with
+ // the exception behavior of Guava cache
+ CaffeinatedGuava.build(builder, appLoader)
Review comment:
Thanks @ben-manes !
We should definitely evaluate alternatives - to understand both the behavior change (particularly if entirely internal to spark, like here) and performance impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805963464
**[Test build #136474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136474/testReport)** for PR 31517 at commit [`8d6ad81`](https://github.com/apache/spark/commit/8d6ad812028950709de4652436b3a58f044ba1a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774966551
**[Test build #135009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135009/testReport)** for PR 31517 at commit [`4761a5b`](https://github.com/apache/spark/commit/4761a5b24637020028f71387e8fecbd4c4f67ba1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872091219
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-892446111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142020/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871972433
**[Test build #140493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140493/testReport)** for PR 31517 at commit [`34d31fd`](https://github.com/apache/spark/commit/34d31fdb2e274d3224c69d257051e03c00f0ca3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774967588
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135009/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883951791
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45898/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806196352
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41058/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-870933109
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r601008117
##########
File path: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala
##########
@@ -431,9 +433,10 @@ class ExecutorSuite extends SparkFunSuite
}
}
- def errorInGuavaCache(e: => Throwable): Throwable = {
- val cache = CacheBuilder.newBuilder()
- .build(new CacheLoader[String, String] {
+ def errorInCaffeinatedGuavaCache(e: => Throwable): Throwable = {
+ val builder = Caffeine.newBuilder()
Review comment:
I found that we need to use `CaffeinatedGuava` with `com.google.common . cache.CacheLoader` to build cache pass this test, this combination is compatible with the exception handling behavior of `Guava` cache
Use `CaffeinatedGuava` with `com.github.benmanes.caffeine.cache.CacheLoader` or use `Caffeine` directly will make this test fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-877965693
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888870938
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ben-manes commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
ben-manes commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r659500929
##########
File path: common/network-shuffle/pom.xml
##########
@@ -58,6 +58,14 @@
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
Review comment:
Could you use Caffeine's `V get(K key, Function<? extends K, ? extends V> mappingFunction)` instead? That should be equivalent but be friendly for method references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806535935
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136507/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872302731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884016511
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141380/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806352279
**[Test build #136498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136498/testReport)** for PR 31517 at commit [`554b5a5`](https://github.com/apache/spark/commit/554b5a5e4b0a0badf16505a0df509cf9aa89bf9c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809955101
**[Test build #136698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136698/testReport)** for PR 31517 at commit [`95d5c2e`](https://github.com/apache/spark/commit/95d5c2e36f4da54e5ad69e212ebdf9e20f32bf1a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-886704373
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r571795620
##########
File path: core/src/test/scala/org/apache/spark/deploy/history/ApplicationCacheSuite.scala
##########
@@ -192,6 +192,7 @@ class ApplicationCacheSuite extends SparkFunSuite with Logging with MockitoSugar
cache.get("2")
cache.get("3")
+ Thread.sleep(5L)
Review comment:
wait data eviction
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r666642523
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##########
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
- } catch (ExecutionException e) {
+ } catch (CompletionException e) {
Review comment:
> Do we have test coverage for this?
Because the `ExecutionException (now is CompletionException)` is re-throw as `RuntimeException`, I need to further check whether it is covered by existing test case
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##########
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
- } catch (ExecutionException e) {
+ } catch (CompletionException e) {
Review comment:
> Do we have test coverage for this?
The `ExecutionException (now is CompletionException)` is re-throw as `RuntimeException`, I need to further check whether it is covered by existing test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873854210
**[Test build #140635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140635/testReport)** for PR 31517 at commit [`e3b81c5`](https://github.com/apache/spark/commit/e3b81c5ee3d2d70eb35e9f8aea04b6716ab5920f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810115867
**[Test build #136709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136709/testReport)** for PR 31517 at commit [`8995d72`](https://github.com/apache/spark/commit/8995d72ecd64ec6407769dbd8545c046898b816b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809571521
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774931027
**[Test build #135011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135011/testReport)** for PR 31517 at commit [`4b49b84`](https://github.com/apache/spark/commit/4b49b84e0c038d286ca09039e774815f4aea7296).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ben-manes commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
ben-manes commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603774600
##########
File path: core/src/test/scala/org/apache/spark/LocalCacheBenchmark.scala
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import scala.util.Random
+
+import com.github.benmanes.caffeine.cache.{CacheLoader => CaffeineCacheLoader, Caffeine}
+import com.github.benmanes.caffeine.guava.CaffeinatedGuava
+import com.google.common.cache.{CacheBuilder, CacheLoader, LoadingCache}
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+
+/**
+ * Benchmark for Guava Cache vs Caffeine.
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt:
+ * bin/spark-submit --class <this class> --jars <spark core test jar>
+ * 2. build/sbt "core/test:runMain <this class>"
+ * 3. generate result:
+ * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "core/test:runMain <this class>"
+ * Results will be written to "benchmarks/KryoBenchmark-results.txt".
+ * }}}
+ */
+object LocalCacheBenchmark extends BenchmarkBase {
+
+ override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+ runBenchmark("Loading Cache") {
+ val size = 10000
+ val parallelism = 8
+ val guavaCacheConcurrencyLevel = 8
+ val dataset = (1 to parallelism)
+ .map(_ => Random.shuffle(List.range(0, size)))
+ .map(list => list.map(i => TestData(i)))
Review comment:
I think your code is fine as is. Maybe just document the simplification? I mostly wanted to let you know since writing a good benchmark is hard, not that you should change it. Your code served its purpose, and you might not get much more out of improving it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884936349
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46009/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884860713
**[Test build #141491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141491/testReport)** for PR 31517 at commit [`0743e3b`](https://github.com/apache/spark/commit/0743e3b6e39f13213f5aa162fbd8bfe1f5c36cf5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871996422
**[Test build #140481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140481/testReport)** for PR 31517 at commit [`47b0bf8`](https://github.com/apache/spark/commit/47b0bf8ccf70cf5ee29b14a352edcce95bf10dda).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884016511
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141380/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883296971
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-890959184
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141935/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-896575303
The benchmark, etc. here look promising. My only concern is that hidden behaviour changes by switching the library. Do they claim compatibility vs Guava at least?
> Does the current use of Caffeine have any negative effects?
So I felt like this had to be investigated first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-804705802
**[Test build #136391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136391/testReport)** for PR 31517 at commit [`ac35393`](https://github.com/apache/spark/commit/ac353937fca565afc441d00d30c3312237c6493f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888870938
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-886743904
**[Test build #141633 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141633/testReport)** for PR 31517 at commit [`f2a656a`](https://github.com/apache/spark/commit/f2a656a94d0d812f4e10434d3a1735a9bf8dd69f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871936677
> Yes, Apache Spark 3.2 will be the first release having Scala 2.13 additionally.
> After branch cut, I'll try to switch the default to Scala 2.13 in master branch.
> Of course, we need the community consensus.
It's an exciting message, haha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-893113204
@holdenk thanks all ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-886661707
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46149/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873905211
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45148/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-804746044
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40975/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806534456
**[Test build #136507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136507/testReport)** for PR 31517 at commit [`89abb61`](https://github.com/apache/spark/commit/89abb61054e3e72455744b85c4d5cff70b8cbd4f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r676824744
##########
File path: core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala
##########
@@ -84,16 +84,18 @@ private[spark] class ReliableCheckpointRDD[T: ClassTag](
}
// Cache of preferred locations of checkpointed files.
- @transient private[spark] lazy val cachedPreferredLocations = CacheBuilder.newBuilder()
- .expireAfterWrite(
- SparkEnv.get.conf.get(CACHE_CHECKPOINT_PREFERRED_LOCS_EXPIRE_TIME).get,
- TimeUnit.MINUTES)
- .build(
- new CacheLoader[Partition, Seq[String]]() {
- override def load(split: Partition): Seq[String] = {
- getPartitionBlockLocations(split)
- }
- })
+ @transient private[spark] lazy val cachedPreferredLocations = {
+ val builder = Caffeine.newBuilder()
+ .expireAfterWrite(
+ SparkEnv.get.conf.get(CACHE_CHECKPOINT_PREFERRED_LOCS_EXPIRE_TIME).get,
+ TimeUnit.MINUTES)
+ val loader = new CacheLoader[Partition, Seq[String]]() {
+ override def load(split: Partition): Seq[String] = {
+ getPartitionBlockLocations(split)
+ }
+ }
+ builder.build[Partition, Seq[String]](loader)
Review comment:
nit: why can't we just use `builder.build[Partition, Seq[String]](getPartitionBlockLocations)`?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -62,21 +61,28 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
+ * @param cause the reason why a `CacheEntry` was removed, it should
+ * always be `SIZE` because `appCache` configured with
+ * `maximumSize` eviction strategy
*/
- override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]): Unit = {
+ override def onRemoval(key: CacheKey, value: CacheEntry,
+ cause: RemovalCause): Unit = {
Review comment:
nit: move this to the previous line?
##########
File path: pom.xml
##########
@@ -492,6 +492,11 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ <version>2.9.1</version>
Review comment:
nit: maybe define `caffeine.version` and put it under `properties`.
##########
File path: core/src/test/scala/org/apache/spark/LocalCacheBenchmark.scala
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.Callable
+
+import scala.concurrent.duration.Duration
+import scala.util.Random
+
+import com.github.benmanes.caffeine.cache.{CacheLoader => CaffeineCacheLoader, Caffeine}
+import com.google.common.cache.{CacheBuilder, CacheLoader}
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+import org.apache.spark.util.ThreadUtils
+
+
+
+/**
+ * Benchmark for Guava Cache vs Caffeine.
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt:
+ * bin/spark-submit --class <this class> --jars <spark core test jar>
+ * 2. build/sbt "core/test:runMain <this class>"
+ * 3. generate result:
+ * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "core/test:runMain <this class>"
+ * Results will be written to "benchmarks/KryoBenchmark-results.txt".
Review comment:
update this
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -159,19 +160,22 @@ class SessionCatalog(
}
private val tableRelationCache: Cache[QualifiedTableName, LogicalPlan] = {
- var builder = CacheBuilder.newBuilder()
+ var builder = Caffeine.newBuilder()
.maximumSize(cacheSize)
if (cacheTTL > 0) {
builder = builder.expireAfterWrite(cacheTTL, TimeUnit.SECONDS)
}
-
- builder.build[QualifiedTableName, LogicalPlan]()
+ // Wrapping as CaffeinatedGuava to be compatible with
Review comment:
Hmm is this comment outdated? I don't see where `CaffeinatedGuava` is used.
##########
File path: core/src/test/scala/org/apache/spark/LocalCacheBenchmark.scala
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.Callable
+
+import scala.concurrent.duration.Duration
+import scala.util.Random
+
+import com.github.benmanes.caffeine.cache.{CacheLoader => CaffeineCacheLoader, Caffeine}
+import com.google.common.cache.{CacheBuilder, CacheLoader}
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+import org.apache.spark.util.ThreadUtils
+
+
Review comment:
nit: extra empty lines.
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
##########
@@ -136,11 +136,14 @@ private[spark] object BlockManagerId {
* The max cache size is hardcoded to 10000, since the size of a BlockManagerId
* object is about 48B, the total memory cost should be below 1MB which is feasible.
*/
- val blockManagerIdCache = CacheBuilder.newBuilder()
- .maximumSize(10000)
- .build(new CacheLoader[BlockManagerId, BlockManagerId]() {
- override def load(id: BlockManagerId) = id
- })
+ val blockManagerIdCache = {
Review comment:
nit: maybe change this to:
```scala
val blockManagerIdCache = Caffeine.newBuilder()
.maximumSize(10000)
.build[BlockManagerId, BlockManagerId](identity)
```
?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -159,19 +160,22 @@ class SessionCatalog(
}
private val tableRelationCache: Cache[QualifiedTableName, LogicalPlan] = {
- var builder = CacheBuilder.newBuilder()
+ var builder = Caffeine.newBuilder()
.maximumSize(cacheSize)
if (cacheTTL > 0) {
builder = builder.expireAfterWrite(cacheTTL, TimeUnit.SECONDS)
}
-
- builder.build[QualifiedTableName, LogicalPlan]()
+ // Wrapping as CaffeinatedGuava to be compatible with
+ // the get(key, valueLoader) API of Guava cache
+ builder.build()
}
/** This method provides a way to get a cached plan. */
def getCachedPlan(t: QualifiedTableName, c: Callable[LogicalPlan]): LogicalPlan = {
- tableRelationCache.get(t, c)
+ tableRelationCache.get(t, new JFunction[QualifiedTableName, LogicalPlan] {
Review comment:
nit: maybe:
```scala
tableRelationCache.get(t, (_: QualifiedTableName) => c.call())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809902830
**[Test build #136687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136687/testReport)** for PR 31517 at commit [`3fad6ef`](https://github.com/apache/spark/commit/3fad6efa16ff78bd3c88a3d27164aa9e14e0f870).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871932774
ok ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809664351
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41239/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888959798
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46331/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876898266
**[Test build #140837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140837/testReport)** for PR 31517 at commit [`68193a9`](https://github.com/apache/spark/commit/68193a9bfd1a1d465e093a8991df9bdc602903c4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805961384
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r659767914
##########
File path: common/network-shuffle/pom.xml
##########
@@ -58,6 +58,14 @@
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
Review comment:
Yes, I'm trying to get rid of the dependence on `com.github.ben-manes.caffeine:guava` in this way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806468142
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806370849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41078/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-890669919
**[Test build #141935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141935/testReport)** for PR 31517 at commit [`706a68d`](https://github.com/apache/spark/commit/706a68dd41c8d67de41355f238b8cb28dccbe243).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806557443
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41092/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887401305
**[Test build #141701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141701/testReport)** for PR 31517 at commit [`87def1a`](https://github.com/apache/spark/commit/87def1aac3b8030584b1696e20d2fe3232643153).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805816667
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136464/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876336974
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45302/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805775757
**[Test build #136458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136458/testReport)** for PR 31517 at commit [`24c361b`](https://github.com/apache/spark/commit/24c361b58825b011bc8aba56c13d62cd1a6e4c1b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r660260370
##########
File path: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala
##########
@@ -452,7 +452,8 @@ class ExecutorSuite extends SparkFunSuite
}
}
- test("SPARK-33587: isFatalError") {
+ // TODO: will fix it later.
Review comment:
ignore this case temporarily due to the difference between guava and caffeine, will fix it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887333011
waiting https://github.com/apache/spark/pull/33533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887245593
**[Test build #141688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141688/testReport)** for PR 31517 at commit [`33f5353`](https://github.com/apache/spark/commit/33f53537bf60fb24a6d252e874d7385e37e41f3b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806510687
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41092/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-890709858
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871334995
There are some busy cases today, we'll try to fix the ignored case tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-892361080
**[Test build #142020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142020/testReport)** for PR 31517 at commit [`81f863f`](https://github.com/apache/spark/commit/81f863ff67d0236f050f4a24e9470c2b1bb7aaff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871006713
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44914/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806354981
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871030376
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44914/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603754412
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -62,21 +64,27 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
- override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]): Unit = {
+ override def onRemoval(key: CacheKey, value: CacheEntry,
+ cause: RemovalCause): Unit = {
metrics.evictionCount.inc()
- val key = rm.getKey
- logDebug(s"Evicting entry ${key}")
- operations.detachSparkUI(key.appId, key.attemptId, rm.getValue().loadedUI.ui)
+ logDebug(s"Evicting entry $key")
+ operations.detachSparkUI(key.appId, key.attemptId, value.loadedUI.ui)
}
}
private val appCache: LoadingCache[CacheKey, CacheEntry] = {
- CacheBuilder.newBuilder()
- .maximumSize(retainedApplications)
- .removalListener(removalListener)
- .build(appLoader)
+ val builder = Caffeine.newBuilder()
+ .maximumSize(retainedApplications)
+ .removalListener(removalListener)
+ // SPARK-34309: Use custom Executor to compatible with
+ // the data eviction behavior of Guava cache
+ .executor((command: Runnable) => command.run())
+ // Wrapping as CaffeinatedGuava to be compatible with
+ // the exception behavior of Guava cache
+ CaffeinatedGuava.build(builder, appLoader)
Review comment:
I think we can remove the `CaffeinatedGuava` in the future, but it would be better to keep it now for two reasons:
- For compatibility of exception handling behavior
For example, there are some differences in the processing of error, when the `get` method throws an `Error`, Guava cache wraps it as an `ExecutionError` and re-throw it, but Caffeine will re-throw `Error` directly. Different exception types and nesting levels may lead to some incompatibilities, such as `SPARK-33587: isFatalError` in `ExecutorSuite`, the test suite will fail if we don't use `CaffeinatedGuava with Guava Cache Loader`.
- For compatibility of API usage
For example, `SessionCatalog` use `V get(K key, Callable<? extends V> valueLoader)` defined in `com.google.common.cache.Cache`, but there is no same interface in `caffeine.cache.Cache`.
In the future, maybe we can use use the `V get(K key, Function<? super K, ? extends V> mappingFunction)` method to replace it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873932372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873881504
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805825896
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774929003
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39592/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-815478471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137047/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810114347
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41290/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-803744853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40906/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r666424023
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##########
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
- } catch (ExecutionException e) {
+ } catch (CompletionException e) {
Review comment:
Can you tell me why we needed to change the exception? Is this just what Caffeine throws instead? Do we have test coverage for this?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -62,21 +61,25 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
- override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]): Unit = {
+ override def onRemoval(key: CacheKey, value: CacheEntry,
+ cause: RemovalCause): Unit = {
metrics.evictionCount.inc()
- val key = rm.getKey
- logDebug(s"Evicting entry ${key}")
- operations.detachSparkUI(key.appId, key.attemptId, rm.getValue().loadedUI.ui)
+ logDebug(s"Evicting entry $key")
+ operations.detachSparkUI(key.appId, key.attemptId, value.loadedUI.ui)
}
}
private val appCache: LoadingCache[CacheKey, CacheEntry] = {
- CacheBuilder.newBuilder()
- .maximumSize(retainedApplications)
- .removalListener(removalListener)
- .build(appLoader)
+ val builder = Caffeine.newBuilder()
+ .maximumSize(retainedApplications)
+ .removalListener(removalListener)
+ // SPARK-34309: Use custom Executor to compatible with
+ // the data eviction behavior of Guava cache
+ .executor((command: Runnable) => command.run())
Review comment:
Is this going to run in the same thread? Is that what the old behaviour would have been?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -62,21 +61,25 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
- override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]): Unit = {
+ override def onRemoval(key: CacheKey, value: CacheEntry,
+ cause: RemovalCause): Unit = {
Review comment:
Maybe mention RemovalCause in the scaladoc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887219770
**[Test build #141679 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141679/testReport)** for PR 31517 at commit [`7b360a7`](https://github.com/apache/spark/commit/7b360a7d577ea379db3487d451b0c7a744d1dc02).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809553552
**[Test build #136657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136657/testReport)** for PR 31517 at commit [`96e6cc8`](https://github.com/apache/spark/commit/96e6cc84546fde64ee5cbfb87a8878427a5b2c88).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809571521
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883296971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-828347005
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42560/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805773577
**[Test build #136456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136456/testReport)** for PR 31517 at commit [`c9f45ae`](https://github.com/apache/spark/commit/c9f45aeabc551041dac385c7a69c6fcf6c1194a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-828310166
**[Test build #138041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138041/testReport)** for PR 31517 at commit [`6146529`](https://github.com/apache/spark/commit/6146529aaf32dbc8f1c6ebcd1dcdf802d706084e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-803730936
**[Test build #136324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136324/testReport)** for PR 31517 at commit [`adc6d92`](https://github.com/apache/spark/commit/adc6d92f1e2d9f02a31198a21d4c306592c23715).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873968232
**[Test build #140638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140638/testReport)** for PR 31517 at commit [`74e2ac3`](https://github.com/apache/spark/commit/74e2ac3259b08cc665239587c88af8312789109d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ExpressionEquals(e: Expression) `
* `case class ExpressionStats(expr: Expression)(var useCount: Int = 1) `
* `case class Average(`
* `case class Sum(`
* `case class SubExprEliminationState(eval: ExprCode, children: Seq[SubExprEliminationState])`
* `case class GetTimestampNTZ(`
* `case class ParseToTimestampNTZ(`
* `trait WatermarkSupport extends SparkPlan `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806336708
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41074/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-828406676
**[Test build #138041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138041/testReport)** for PR 31517 at commit [`6146529`](https://github.com/apache/spark/commit/6146529aaf32dbc8f1c6ebcd1dcdf802d706084e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603773282
##########
File path: core/src/test/scala/org/apache/spark/LocalCacheBenchmark.scala
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import scala.util.Random
+
+import com.github.benmanes.caffeine.cache.{CacheLoader => CaffeineCacheLoader, Caffeine}
+import com.github.benmanes.caffeine.guava.CaffeinatedGuava
+import com.google.common.cache.{CacheBuilder, CacheLoader, LoadingCache}
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+
+/**
+ * Benchmark for Guava Cache vs Caffeine.
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt:
+ * bin/spark-submit --class <this class> --jars <spark core test jar>
+ * 2. build/sbt "core/test:runMain <this class>"
+ * 3. generate result:
+ * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "core/test:runMain <this class>"
+ * Results will be written to "benchmarks/KryoBenchmark-results.txt".
+ * }}}
+ */
+object LocalCacheBenchmark extends BenchmarkBase {
+
+ override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+ runBenchmark("Loading Cache") {
+ val size = 10000
+ val parallelism = 8
+ val guavaCacheConcurrencyLevel = 8
+ val dataset = (1 to parallelism)
+ .map(_ => Random.shuffle(List.range(0, size)))
+ .map(list => list.map(i => TestData(i)))
Review comment:
Thank you for your advice. I think we should avoid introducing more dependencies, so I'll try to implement this data generator in spark code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809479812
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888943405
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46331/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873902990
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805519253
**[Test build #136432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136432/testReport)** for PR 31517 at commit [`b568fcc`](https://github.com/apache/spark/commit/b568fcc00ddc52580c2b7212bb81f1942f0a748a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809998533
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41280/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806421936
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136498/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-881278083
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45648/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805737047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136442/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809664307
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41239/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-890685532
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805811368
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41037/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ben-manes commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
ben-manes commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r603579741
##########
File path: core/src/test/scala/org/apache/spark/LocalCacheBenchmark.scala
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import scala.util.Random
+
+import com.github.benmanes.caffeine.cache.{CacheLoader => CaffeineCacheLoader, Caffeine}
+import com.github.benmanes.caffeine.guava.CaffeinatedGuava
+import com.google.common.cache.{CacheBuilder, CacheLoader, LoadingCache}
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+
+/**
+ * Benchmark for Guava Cache vs Caffeine.
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt:
+ * bin/spark-submit --class <this class> --jars <spark core test jar>
+ * 2. build/sbt "core/test:runMain <this class>"
+ * 3. generate result:
+ * SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "core/test:runMain <this class>"
+ * Results will be written to "benchmarks/KryoBenchmark-results.txt".
+ * }}}
+ */
+object LocalCacheBenchmark extends BenchmarkBase {
+
+ override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+ runBenchmark("Loading Cache") {
+ val size = 10000
+ val parallelism = 8
+ val guavaCacheConcurrencyLevel = 8
+ val dataset = (1 to parallelism)
+ .map(_ => Random.shuffle(List.range(0, size)))
+ .map(list => list.map(i => TestData(i)))
Review comment:
This distribution is uniformly distributed with only single key overlaps. This means that there are not hot and cold entries, e.g. random eviction has an optimal hit rate. In reality, some entries will be used much more often and follows a power law curve.
That is fairly generous distribution for a cache like guava, which uses coarse locking of multiple hash tables. That way the access distribution matches the hash distribution, so ideally spread across all of the locks. In reality, while the hash distribution will be uniform the access distribution is not so a lock holding hot entries will be used much more frequently.
In Caffeine's [benchmarks](https://github.com/ben-manes/caffeine/blob/master/caffeine/src/jmh/java/com/github/benmanes/caffeine/cache/GetPutBenchmark.java), it uses a scrambled Zipfian distribution (YCSB's generator). That would show an even larger speedup.
More just an fyi that your benchmarks are conservative and you may see a larger gain. Of course, if the caches are not a bottleneck you might not see any benefit except if the eviction policy improves the hit rates in your workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809382831
**[Test build #136657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136657/testReport)** for PR 31517 at commit [`96e6cc8`](https://github.com/apache/spark/commit/96e6cc84546fde64ee5cbfb87a8878427a5b2c88).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805473353
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805919425
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41042/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805780469
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873857147
**[Test build #140635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140635/testReport)** for PR 31517 at commit [`e3b81c5`](https://github.com/apache/spark/commit/e3b81c5ee3d2d70eb35e9f8aea04b6716ab5920f).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887199134
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46193/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888598588
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46296/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887220339
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141679/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872003267
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-876967340
**[Test build #140837 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140837/testReport)** for PR 31517 at commit [`68193a9`](https://github.com/apache/spark/commit/68193a9bfd1a1d465e093a8991df9bdc602903c4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805781960
**[Test build #136459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136459/testReport)** for PR 31517 at commit [`2b3072e`](https://github.com/apache/spark/commit/2b3072e290c765b6c74e898839a181fb3b64a693).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806397562
**[Test build #136504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136504/testReport)** for PR 31517 at commit [`f488a3b`](https://github.com/apache/spark/commit/f488a3be37bcf1b42c6a4f34b8bcc804007bf20b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-892411356
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46531/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883180832
**[Test build #141309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141309/testReport)** for PR 31517 at commit [`5423f36`](https://github.com/apache/spark/commit/5423f36d7933c05548b9570ee121c7cc76a90c9b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887178875
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46192/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810253598
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136709/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809902096
**[Test build #136687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136687/testReport)** for PR 31517 at commit [`3fad6ef`](https://github.com/apache/spark/commit/3fad6efa16ff78bd3c88a3d27164aa9e14e0f870).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888878285
**[Test build #141818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141818/testReport)** for PR 31517 at commit [`5a75b2c`](https://github.com/apache/spark/commit/5a75b2cf1c70d9317bec6a65bb4211027ff290d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888820990
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46319/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887180729
**[Test build #141679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141679/testReport)** for PR 31517 at commit [`7b360a7`](https://github.com/apache/spark/commit/7b360a7d577ea379db3487d451b0c7a744d1dc02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883942834
@holdenk c6b9dc6 merge with master and waiting for Jenkins build,
It seems that GA is unstable, and many pr have similar failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-888839792
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46319/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884993242
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141491/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883888248
**[Test build #141380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141380/testReport)** for PR 31517 at commit [`c6b9dc6`](https://github.com/apache/spark/commit/c6b9dc69d20caf4732ee85872e8b103cebe8ba0f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884903611
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46009/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805466009
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806370494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809998555
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41280/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805857975
**[Test build #136467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136467/testReport)** for PR 31517 at commit [`32d9ce0`](https://github.com/apache/spark/commit/32d9ce0ca259fe4fa61424aa3288a62232bc4164).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805454967
**[Test build #136432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136432/testReport)** for PR 31517 at commit [`b568fcc`](https://github.com/apache/spark/commit/b568fcc00ddc52580c2b7212bb81f1942f0a748a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774903911
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39589/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-886745559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141633/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883180832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806370494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806491358
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806317925
**[Test build #136492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136492/testReport)** for PR 31517 at commit [`5eb75fa`](https://github.com/apache/spark/commit/5eb75fa9a4c4382dc44169c36a1161b228479b31).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-803248133
Gentle ping, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774997260
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135006/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806535935
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136507/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805646499
**[Test build #136450 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136450/testReport)** for PR 31517 at commit [`97c3c74`](https://github.com/apache/spark/commit/97c3c741b619ea69afaf89aeb3975f91ef3eb2ee).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887466253
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-892361080
**[Test build #142020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142020/testReport)** for PR 31517 at commit [`81f863f`](https://github.com/apache/spark/commit/81f863ff67d0236f050f4a24e9470c2b1bb7aaff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #31517:
URL: https://github.com/apache/spark/pull/31517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r600136193
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubExprEvaluationRuntimeSuite.scala
##########
@@ -45,6 +45,8 @@ class SubExprEvaluationRuntimeSuite extends SparkFunSuite {
val proxy3 = ExpressionProxy(Literal(3), 2, runtime)
proxy3.eval()
+ // wait data Eviction
Review comment:
@ben-manes I have another question, here I add `Thread.sleep(5L)` to wait data eviction, because I find that the data eviction has some delay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-775117864
Just out of curiosity, how much performance improvement we can get by using Caffeine? Do we have a rough number?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805780469
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872126766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140493/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r666639201
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##########
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
- } catch (ExecutionException e) {
+ } catch (CompletionException e) {
Review comment:
> Can you tell me why we needed to change the exception? Is this just what Caffeine throws instead?
`com.github.benmanes.caffeine.cache.LoadingCache#get` method throw `CompletionException` if a checked exception was thrown while loading the value.
`com.google.common.cache.LoadingCache#get` method throw `ExecutionException` if a checked exception was thrown while loading the value.
So the exception type here has changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-873932372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805564609
**[Test build #136442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136442/testReport)** for PR 31517 at commit [`ca9d58d`](https://github.com/apache/spark/commit/ca9d58d1a9a94694509ede65e87471e794bf307c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-774906502
**[Test build #135006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135006/testReport)** for PR 31517 at commit [`0c5382a`](https://github.com/apache/spark/commit/0c5382af0a54c5db8cf9ffee6a7a5040be5cb1c7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-872077840
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805473353
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-884991261
**[Test build #141491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141491/testReport)** for PR 31517 at commit [`0743e3b`](https://github.com/apache/spark/commit/0743e3b6e39f13213f5aa162fbd8bfe1f5c36cf5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Observation(name: String) `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-877991274
**[Test build #140893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140893/testReport)** for PR 31517 at commit [`ee05314`](https://github.com/apache/spark/commit/ee053145c3a45a9f63fdb37c8c2b246498067376).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public final class SparkOutOfMemoryError extends OutOfMemoryError implements SparkThrowable `
* ` throw new IllegalArgumentException(s\"Cannot find error class '$errorClass'\"))`
* `class SparkArithmeticException(errorClass: String, messageParameters: Array[String])`
* ` class RemoteBlockDownloadFileManager(`
* ` class RocksDBStateStore(lastVersion: Long) extends StateStore `
* ` class StateEncoder `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806529210
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136504/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-887500926
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46215/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-803723448
Sorry to forget this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805813633
**[Test build #136464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136464/testReport)** for PR 31517 at commit [`d1315c3`](https://github.com/apache/spark/commit/d1315c3b2fe060654ce4c1881c6d58be97835cc4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r658950185
##########
File path: common/network-shuffle/pom.xml
##########
@@ -58,6 +58,14 @@
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
Review comment:
Could we avoid this since it depends on Guava and guava conflicts can be a pain to resolve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809431699
**[Test build #136661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136661/testReport)** for PR 31517 at commit [`7028ffa`](https://github.com/apache/spark/commit/7028ffa3f753d2a95f53a5d0cd5fca82dfc237b1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r666673952
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##########
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
- } catch (ExecutionException e) {
+ } catch (CompletionException e) {
Review comment:
@holdenk `testFetchWrongExecutor` and `testFetchNonexistent` in `ExternalBlockHandlerSuite` already cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ben-manes commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
ben-manes commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r599720425
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##########
@@ -58,24 +58,26 @@ private[history] class ApplicationCache(
}
- private val removalListener = new RemovalListener[CacheKey, CacheEntry] {
+ private val cacheWriter = new CacheWriter[CacheKey, CacheEntry] {
Review comment:
@LuciferYang This can be accomplished by using `cache.asMap().computeIfPresent` instead of `invalidate`. Then you may callback manually within the atomic operation and return `null` to remove the entry.
`CacheWriter` tried to intercept but it was confusing by missing some cases. For example should a `invalidate(key)` call the writer if the entry is absent? It does not, but in a multi-layer cache one might expect to propagate the removal even if absent. These quirks led to replacing it with explicit callbacks via compute for manual operations and `evictionListener` for automatic ones. An invalidation is a manual operation, so it requires a map compute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-886613290
**[Test build #141633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141633/testReport)** for PR 31517 at commit [`f2a656a`](https://github.com/apache/spark/commit/f2a656a94d0d812f4e10434d3a1735a9bf8dd69f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-806393925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-871935280
@dongjoon-hyun A question unrelated to this pr, do we have plans to make Scala 2.13 as default？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810150295
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r659495725
##########
File path: common/network-shuffle/pom.xml
##########
@@ -58,6 +58,14 @@
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>caffeine</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
Review comment:
This is a good idea, but there are some interface differences between Caffeine and Guava Cache, and spark just uses these interfaces. For example, `SessionCatalog` uses `V get(K key, Callable<? extends V> valueLoader)` method only define in Guava Cache API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-805775757
**[Test build #136458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136458/testReport)** for PR 31517 at commit [`24c361b`](https://github.com/apache/spark/commit/24c361b58825b011bc8aba56c13d62cd1a6e4c1b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #31517:
URL: https://github.com/apache/spark/pull/31517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-890669919
**[Test build #141935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141935/testReport)** for PR 31517 at commit [`706a68d`](https://github.com/apache/spark/commit/706a68dd41c8d67de41355f238b8cb28dccbe243).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-809998555
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41280/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-810150295
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31517:
URL: https://github.com/apache/spark/pull/31517#issuecomment-883180832
**[Test build #141309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141309/testReport)** for PR 31517 at commit [`5423f36`](https://github.com/apache/spark/commit/5423f36d7933c05548b9570ee121c7cc76a90c9b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r677082323
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -159,19 +160,22 @@ class SessionCatalog(
}
private val tableRelationCache: Cache[QualifiedTableName, LogicalPlan] = {
- var builder = CacheBuilder.newBuilder()
+ var builder = Caffeine.newBuilder()
.maximumSize(cacheSize)
if (cacheTTL > 0) {
builder = builder.expireAfterWrite(cacheTTL, TimeUnit.SECONDS)
}
-
- builder.build[QualifiedTableName, LogicalPlan]()
+ // Wrapping as CaffeinatedGuava to be compatible with
+ // the get(key, valueLoader) API of Guava cache
+ builder.build()
}
/** This method provides a way to get a cached plan. */
def getCachedPlan(t: QualifiedTableName, c: Callable[LogicalPlan]): LogicalPlan = {
- tableRelationCache.get(t, c)
+ tableRelationCache.get(t, new JFunction[QualifiedTableName, LogicalPlan] {
Review comment:
d69df8e fixed these comments. The change of here need to see if it can be compiled
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -159,19 +160,22 @@ class SessionCatalog(
}
private val tableRelationCache: Cache[QualifiedTableName, LogicalPlan] = {
- var builder = CacheBuilder.newBuilder()
+ var builder = Caffeine.newBuilder()
.maximumSize(cacheSize)
if (cacheTTL > 0) {
builder = builder.expireAfterWrite(cacheTTL, TimeUnit.SECONDS)
}
-
- builder.build[QualifiedTableName, LogicalPlan]()
+ // Wrapping as CaffeinatedGuava to be compatible with
+ // the get(key, valueLoader) API of Guava cache
+ builder.build()
}
/** This method provides a way to get a cached plan. */
def getCachedPlan(t: QualifiedTableName, c: Callable[LogicalPlan]): LogicalPlan = {
- tableRelationCache.get(t, c)
+ tableRelationCache.get(t, new JFunction[QualifiedTableName, LogicalPlan] {
Review comment:
d69df8e fixed these comments. The change of there need to see if it can be compiled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org