You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sahil Takiar <ta...@gmail.com> on 2018/10/01 15:19:40 UTC
Re: Review Request 68474: HIVE-20440: Create better cache eviction
policy for SmallTableCache
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68474/#review209130
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Line 60 (original), 69 (patched)
<https://reviews.apache.org/r/68474/#comment293420>
keep the explicit cache method and call it in `MapJoinOperator#closeOp`. This way when a task finishes, we still keep the small table around for at least 30 seconds, which gives any tasks scheduled in the future a chance to re-use the small table.
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 75 (patched)
<https://reviews.apache.org/r/68474/#comment293419>
can u add some javadocs to this class explaining what it is doing
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 82 (patched)
<https://reviews.apache.org/r/68474/#comment293416>
rename to something like `cleanupService`
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 90 (patched)
<https://reviews.apache.org/r/68474/#comment293417>
nit: make `INTEGER_ONE` a static import
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 91 (patched)
<https://reviews.apache.org/r/68474/#comment293415>
"SmallTableCache maintenance thread" -> "SmallTableCache Cleanup Thread"
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 117 (patched)
<https://reviews.apache.org/r/68474/#comment293418>
replace with `cacheL1.get(key, valueLoader)` where `valueLoader` loads from `cacheL2`
- Sahil Takiar
On Sept. 19, 2018, 11:14 p.m., Antal Sinkovits wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68474/
> -----------------------------------------------------------
>
> (Updated Sept. 19, 2018, 11:14 p.m.)
>
>
> Review request for hive, Naveen Gangam, Sahil Takiar, Adam Szita, and Xuefu Zhang.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> I've modified the SmallTableCache to use guava cache, with soft references.
> By using a value loader, I've also eliminated the synchronization on the intern-ed string of the path.
>
>
> Diffs
> -----
>
> ql/pom.xml d73deba440702ec39fc5610df28e0fe54baef025
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java cf27e92bafdc63096ec0fa8c3106657bab52f370
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java 3293100af96dc60408c53065fa89143ead98f818
> ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSmallTableCache.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/68474/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Antal Sinkovits
>
>