You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openjpa.apache.org by "Dimitrios Iosifidis (Created) (JIRA)" <ji...@apache.org> on 2011/11/22 15:26:40 UTC
[jira] [Created] (OPENJPA-2078) Using external datacache - poor
performance when trying to fetch a large resultSet
Using external datacache - poor performance when trying to fetch a large resultSet
----------------------------------------------------------------------------------
Key: OPENJPA-2078
URL: https://issues.apache.org/jira/browse/OPENJPA-2078
Project: OpenJPA
Issue Type: Question
Components: datacache, performance
Affects Versions: 2.1.0
Environment: Mysql, memcached, SLES 11
Reporter: Dimitrios Iosifidis
Priority: Critical
In our application, we have extended the AbstractDataCache/AbstractQueryCache in order to introduce our own caching implementation using Memcached.
However, during testing it has been found that performance has dramatically degraded. Analysis has shown that the main reason for this is the fact that openjpa calls the
protected DataCachePCData getInternal(final Object arg0)
method instead (probably) of the:
Map<Object, DataCachePCData> getAll(List<Object> arg0)
For a result set of 40000 records for example, 40000 get requests are issued to the external cache, resulting in poor performance. Instead, if the getAll(..) method (overrided from AbstractDataCache) was to be called, then the cache would have a decent performance (as a consequence of a single request).
So, my question consists of the following:
1) Is there a relevance between query cache and data cache? The query I am experimenting on is like:
SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName
for which I get at the console output:
"WARN [Runtime] Query "SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName" is removed from cache excluded permanently. Query "SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName" is not cached because its result is not obtained by executing a select statement. This can happen if the query was evaluated in-memory. The result was provided by org.apache.openjpa.datacache.QueryCacheStoreQuery$CachingResultObjectProvider. "
This means subsequently that this type of query cannot benefit from caching techniques?
2) Is there a way to get all ids of the PO's from openjpa that consist the resultlist of a specific query?
3) In this particular example, enabling the default cache (openJPA's):
<property name="openjpa.DataCache" value="true"/>
<property name="openjpa.QueryCache" value="true"/>
seems to work fine (in respect of performance)
The above queries are issued two times, but only one reached the database. Taking into account that the same warning is displayed (Query "SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName" is removed from cache excluded permanently.....) how this works?
4) Is there a mechanism to trigger using the "bulk" getAll, loadAll() etc methods?
Any hints are highly appreciated!
Thank you in advance,
Dimitris.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OPENJPA-2078) Using external datacache - poor
performance when trying to fetch a large resultSet
Posted by "Rick Curtis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OPENJPA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155201#comment-13155201 ]
Rick Curtis commented on OPENJPA-2078:
--------------------------------------
First off, we'd prefer to have you wait until we have identified that a code change is necessary before opening a JIRA. For future reference, could you use the users/dev mailing lists to ask questions?
Now on to answering your questions.
> 1) Is there a relevance between query cache and data cache?
Yes, the openja.QueryCache(QueryResultsCache) can be thought of as a mapping of SQL (and params) to a collection of oids. While it is possible to use the query cache without a data cache, that usage doesn't make much sense for most users.
> This means subsequently that this type of query cannot benefit from caching techniques?
Honestly, I'm not 100% on the meaning of that message, but I'm fairly certain it is a noisy message coming from another one of the internal OpenJPA query caches.... I think it can be safely ignored.
> 2) Is there a way to get all ids of the PO's from openjpa that consist the resultlist of a specific query?
Yes, that is how the data is stored in the QueryCache. [key] -> Collection<PO>
> 3) In this particular example, enabling the default cache (openJPA's):
I believe this statement supports my previous comment that the indicated message is just noise.
> 4) Is there a mechanism to trigger using the "bulk" getAll, loadAll() etc methods?
Unfortunately there isn't... but I would think that we could do something along these lines. As you may have noticed, OpenJPA is quite bad at it's cache usage. We issue lots and lots of remote calls.
Hopefully this answers a majority of your questions.
Thanks,
Rick
> Using external datacache - poor performance when trying to fetch a large resultSet
> ----------------------------------------------------------------------------------
>
> Key: OPENJPA-2078
> URL: https://issues.apache.org/jira/browse/OPENJPA-2078
> Project: OpenJPA
> Issue Type: Question
> Components: datacache, performance
> Affects Versions: 2.1.0
> Environment: Mysql, memcached, SLES 11
> Reporter: Dimitrios Iosifidis
> Priority: Critical
>
> In our application, we have extended the AbstractDataCache/AbstractQueryCache in order to introduce our own caching implementation using Memcached.
> However, during testing it has been found that performance has dramatically degraded. Analysis has shown that the main reason for this is the fact that openjpa calls the
>
> protected DataCachePCData getInternal(final Object arg0)
> method instead (probably) of the:
> Map<Object, DataCachePCData> getAll(List<Object> arg0)
> For a result set of 40000 records for example, 40000 get requests are issued to the external cache, resulting in poor performance. Instead, if the getAll(..) method (overrided from AbstractDataCache) was to be called, then the cache would have a decent performance (as a consequence of a single request).
> So, my question consists of the following:
> 1) Is there a relevance between query cache and data cache? The query I am experimenting on is like:
> SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName
> for which I get at the console output:
> "WARN [Runtime] Query "SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName" is removed from cache excluded permanently. Query "SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName" is not cached because its result is not obtained by executing a select statement. This can happen if the query was evaluated in-memory. The result was provided by org.apache.openjpa.datacache.QueryCacheStoreQuery$CachingResultObjectProvider. "
> This means subsequently that this type of query cannot benefit from caching techniques?
> 2) Is there a way to get all ids of the PO's from openjpa that consist the resultlist of a specific query?
> 3) In this particular example, enabling the default cache (openJPA's):
> <property name="openjpa.DataCache" value="true"/>
> <property name="openjpa.QueryCache" value="true"/>
> seems to work fine (in respect of performance)
> The above queries are issued two times, but only one reached the database. Taking into account that the same warning is displayed (Query "SELECT user FROM UserPO AS user WHERE user.domain.fname = :domainName" is removed from cache excluded permanently.....) how this works?
> 4) Is there a mechanism to trigger using the "bulk" getAll, loadAll() etc methods?
> Any hints are highly appreciated!
> Thank you in advance,
> Dimitris.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira