You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/01/15 13:35:51 UTC
[GitHub] [carbondata] Indhumathi27 opened a new pull request #3580: [WIP]
Support TimeBased Cache expiration using Guava Cache
Indhumathi27 opened a new pull request #3580: [WIP] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r369374280
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCacheMap;
Review comment:
Currently, using guava cache, we can do only time-based or size-based eviction for all values(not specific to table level) loaded to CarbonLruCache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-577554016
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1754/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371041664
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCacheMap;
Review comment:
@jackylk , @ravipesala : what do you think ?
@Indhumathi27 : I still think we must have **entry level expiry**, just removing all the cache after certain time is not so useful as cache need to be loaded again ?
check this guava issue for the same ([here](https://github.com/google/guava/issues/1203))
I just checked on internet and I found 2 alternatives (under apache license) that supports entry level expiry.
a. [caffeine](https://github.com/ben-manes/caffeine) -- This is on top of guava as they support guava adaptor
b. [expiringmap](https://github.com/jhalterman/expiringmap) -- This is widely used my many companies in their project. see [here](https://github.com/jhalterman/expiringmap/wiki/Who's-Using-ExpiringMap). I guess we can use this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371613959
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCacheMap;
Review comment:
@ajantha-bhat @akashrn5 ok. I will check those caching libraries and will raise new PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-577523348
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1752/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xuchuanyin commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
xuchuanyin commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r368768906
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -95,9 +98,17 @@ public CarbonLRUCache(String propertyName, String defaultPropertyName) {
* initialize lru cache
*/
private void initCache() {
+ // get duration for cache expiration if configured
+ long duration = CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT;
+ String timeBasedExpiration = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES);
+ if (null != timeBasedExpiration) {
+ duration = Long.parseLong(timeBasedExpiration);
Review comment:
forget to validate the content?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r369374454
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -95,9 +98,17 @@ public CarbonLRUCache(String propertyName, String defaultPropertyName) {
* initialize lru cache
*/
private void initCache() {
+ // get duration for cache expiration if configured
+ long duration = CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT;
+ String timeBasedExpiration = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES);
+ if (null != timeBasedExpiration) {
+ duration = Long.parseLong(timeBasedExpiration);
+ }
+ // initialise guava cache with time based expiration
lruCacheMap =
- new LinkedHashMap<String, Cacheable>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE, 1.0f,
- true);
+ CacheBuilder.newBuilder().initialCapacity(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE)
+ .expireAfterWrite(duration, TimeUnit.MINUTES).build();
Review comment:
Size-based eviction can be supported later if needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on issue #3580: [WIP] Support
TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on issue #3580: [WIP] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-575119953
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r368259071
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCacheMap;
Review comment:
There are scenarios where cache of some table need to be maintained and other tables can be dropped. **It will be good if we support table level expiry.** and not clear all the cache when expire.
Drop meta cache DML is already supporting table level dropping , but it is manual work now. If this can be handled by guava, it is great
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371007080
##########
File path: core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
##########
@@ -1894,4 +1898,35 @@ public static String getDataMapStorageProvider() {
}
return provider.toUpperCase();
}
+
+ /**
+ * This method validates the duration for carbon cache expiration
+ */
+ private void validateDurationForCacheExpiration() {
+ String defaultValue = Long.toString(
+ CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT);
+ long duration;
+ try {
+ duration = Long.parseLong(
Review comment:
rename to carbonCacheExpireDuration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xuchuanyin commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
xuchuanyin commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r368769593
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -95,9 +98,17 @@ public CarbonLRUCache(String propertyName, String defaultPropertyName) {
* initialize lru cache
*/
private void initCache() {
+ // get duration for cache expiration if configured
+ long duration = CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT;
+ String timeBasedExpiration = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES);
+ if (null != timeBasedExpiration) {
+ duration = Long.parseLong(timeBasedExpiration);
+ }
+ // initialise guava cache with time based expiration
lruCacheMap =
Review comment:
better to optimize this variable's name -- `lruCache` is enough
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3580:
[WIP][CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3580: [WIP][CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-575149816
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1663/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371007560
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCache;
Review comment:
since now least recently used cache is changed to time based, does this variable name still makes sense? shall we give a better one to depict the time based cache implementation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r369371985
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -95,9 +98,17 @@ public CarbonLRUCache(String propertyName, String defaultPropertyName) {
* initialize lru cache
*/
private void initCache() {
+ // get duration for cache expiration if configured
+ long duration = CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT;
+ String timeBasedExpiration = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES);
+ if (null != timeBasedExpiration) {
+ duration = Long.parseLong(timeBasedExpiration);
Review comment:
Configuration is already validated in `core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java`. Please check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371041664
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCacheMap;
Review comment:
@jackylk , @ravipesala : what do you think ?
@Indhumathi27 : I still think we must have **entry level expiry**, just removing all the cache after certain time is not so useful as cache need to be loaded again ?
check this guava issue for the same ([here](https://github.com/google/guava/issues/1203))
I just checked on internet and I found 2 alternatives (under apache license) that supports entry level expiry.
a. [caffeine](https://github.com/ben-manes/caffeine) -- This is on top of guava as they support guava adaptor
b. [expiringmap](https://github.com/jhalterman/expiringmap) -- This is widely used by many companies in their project. see [here](https://github.com/jhalterman/expiringmap/wiki/Who's-Using-ExpiringMap). I guess we can use this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-575447231
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1672/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r369945607
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -95,9 +98,17 @@ public CarbonLRUCache(String propertyName, String defaultPropertyName) {
* initialize lru cache
*/
private void initCache() {
+ // get duration for cache expiration if configured
+ long duration = CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT;
+ String timeBasedExpiration = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES);
+ if (null != timeBasedExpiration) {
+ duration = Long.parseLong(timeBasedExpiration);
+ }
+ // initialise guava cache with time based expiration
lruCacheMap =
Review comment:
changed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3580: [WIP] Support
TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3580: [WIP] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-574696831
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1646/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3580: [WIP] Support
TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3580: [WIP] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#issuecomment-574939662
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1647/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xuchuanyin commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
xuchuanyin commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r368769292
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -95,9 +98,17 @@ public CarbonLRUCache(String propertyName, String defaultPropertyName) {
* initialize lru cache
*/
private void initCache() {
+ // get duration for cache expiration if configured
+ long duration = CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT;
+ String timeBasedExpiration = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES);
+ if (null != timeBasedExpiration) {
+ duration = Long.parseLong(timeBasedExpiration);
+ }
+ // initialise guava cache with time based expiration
lruCacheMap =
- new LinkedHashMap<String, Cacheable>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE, 1.0f,
- true);
+ CacheBuilder.newBuilder().initialCapacity(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE)
+ .expireAfterWrite(duration, TimeUnit.MINUTES).build();
Review comment:
expire after write or access?
besides, guava also support size based cache, will it be supported later?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r369387237
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -327,15 +309,15 @@ public Cacheable get(String key) {
*/
public void clear() {
synchronized (lruCacheMap) {
- for (Cacheable cachebleObj : lruCacheMap.values()) {
+ for (Cacheable cachebleObj : lruCacheMap.asMap().values()) {
Review comment:
the above code is for invalidating Cacheable objects. For LRUCache, cleanUp method is used for clearing the cache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 closed pull request #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
Indhumathi27 closed pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xuchuanyin commented on a change in pull request
#3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
xuchuanyin commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r368770941
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -327,15 +309,15 @@ public Cacheable get(String key) {
*/
public void clear() {
synchronized (lruCacheMap) {
- for (Cacheable cachebleObj : lruCacheMap.values()) {
+ for (Cacheable cachebleObj : lruCacheMap.asMap().values()) {
Review comment:
why not use method `invalidateAll()` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371065288
##########
File path: core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##########
@@ -43,10 +45,11 @@
private static final Logger LOGGER =
LogServiceFactory.getLogService(CarbonLRUCache.class.getName());
/**
- * Map that will contain key as table unique name and value as cache Holder
+ * Guava cache that holds key as table unique name and value as cache Holder
* object
*/
- private Map<String, Cacheable> lruCacheMap;
+ private Cache<String, Cacheable> lruCacheMap;
Review comment:
@Indhumathi27 i agree with @ajantha-bhat , may we need to keep option, or give choices to user and we should have implementations based on that like time based, size based.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3580:
[CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3580: [CARBONDATA-3665] Support TimeBased Cache expiration using Guava Cache
URL: https://github.com/apache/carbondata/pull/3580#discussion_r371007320
##########
File path: core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
##########
@@ -1894,4 +1898,35 @@ public static String getDataMapStorageProvider() {
}
return provider.toUpperCase();
}
+
+ /**
+ * This method validates the duration for carbon cache expiration
+ */
+ private void validateDurationForCacheExpiration() {
+ String defaultValue = Long.toString(
+ CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT);
+ long duration;
+ try {
+ duration = Long.parseLong(
+ carbonProperties.getProperty(CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES));
+ if (duration < 0 || duration == 0) {
+ LOGGER.info("using default value of carbon.lru.cache.expiration.duration.in.minutes = "
+ + CarbonCommonConstants.CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES_DEFAULT);
+ carbonProperties
+ .setProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB,
+ defaultValue);
+ } else {
+ LOGGER.info("using carbon.lru.cache.expiration.duration.in.minutes = " + duration);
+ carbonProperties
+ .setProperty(CARBON_LRU_CACHE_EXPIRATION_DURATION_IN_MINUTES,
+ Long.toString(duration));
+ }
+ } catch (Exception e) {
Review comment:
In this method, only NumberFormatException will be thrown, i dont see any other type of exception. If not instead of generalised exception, use NumberFormatException.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services