You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Peter Ondruška <pe...@kaibo.eu> on 2017/07/06 14:14:24 UTC
Page cache sizing
Dear all,
how does derby.storage.pageCacheSize parameter (
https://db.apache.org/derby/docs/10.13/ref/rrefproper81359.html) work with
database that has multiple page sizes--tables with default 4096 bytes and
tables with long/blob of 32768 byte pages?
Thanks,
--
Peter Ondruška
--
kaibo, s.r.o., ID 28435036, registered with the commercial register
administered by the Municipal Court in Prague, section C, insert 141269.
Registered office and postal address: kaibo, s.r.o., Kališnická 379/10,
Prague 3, 130 00, Czech Republic.
https://www.kaibo.eu
Re: Page cache sizing
Posted by Peter Ondruška <pe...@kaibo.eu>.
Thanks Bryan,
I have done different kind of test so far and this is my observation:
1. set network server JVM heap size to 512 M
2. set parameter derby.storage.pageCacheSize=16384
3. booted a database with mixed block sizes (default 4 K, and large 32 K)
4. after some time I get OOM when conneting network client, this OOM is
reported back from network server, see * and ** (tried to backup as well)
5. only changed parameter derby.storage.pageCacheSize=4096
6. no more OOM
Conclusion seems to be that pageCacheSize may be based really on just the
number of cached blocks (as documented) and if they are mixed you better
size based on 32 K size. If this is confirmed by the developers I would
suggest ammending documentation with such note because some objects may
automatically use 32 K page (LOBs) and can cause OOM suddenly if
pageCacheSize was set for 4 K block size. But ideally a separate
pageCacheSize setting would be great for large/non-standard block size(s).
*
Sun Jul 09 00:24:20 CEST 2017 Thread[DRDAConnThread_3,5,main] (DATABASE =
store), (DRDAID = {233}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 01:02:48 CEST 2017 Thread[DRDAConnThread_7,5,main] (DATABASE =
store), (DRDAID = {266}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 01:24:11 CEST 2017 Thread[DRDAConnThread_6,5,main] (DATABASE =
store), (DRDAID = {283}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 02:24:12 CEST 2017 Thread[DRDAConnThread_7,5,main] (DATABASE =
store), (DRDAID = {332}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 03:24:11 CEST 2017 Thread[DRDAConnThread_4,5,main] (DATABASE =
store), (DRDAID = {381}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 04:24:10 CEST 2017 Thread[DRDAConnThread_10,5,main] (DATABASE =
store), (DRDAID = {430}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 04:30:30 CEST 2017 Thread[DRDAConnThread_9,5,main] (DATABASE =
store), (DRDAID = {439}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 04:30:31 CEST 2017 Thread[DRDAConnThread_9,5,main] (DATABASE =
store), (DRDAID = {440}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 05:24:11 CEST 2017 Thread[DRDAConnThread_3,5,main] (DATABASE =
store), (DRDAID = {481}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 06:24:12 CEST 2017 Thread[DRDAConnThread_2,5,main] (DATABASE =
store), (DRDAID = {530}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 07:24:12 CEST 2017 Thread[DRDAConnThread_10,5,main] (DATABASE =
store), (DRDAID = {579}), Connection refused : java.lang.OutOfMemoryError
Sun Jul 09 07:24:45 CEST 2017 Thread[DRDAConnThread_3,5,main] (DATABASE =
ejbtimer), (DRDAID = {582}), Connection refused : java.lang.OutOfMemoryError
**
ERROR XJ001: Java exception: 'java.lang.OutOfMemoryError: Java heap space'.
at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at
org.apache.derby.iapi.error.StandardException.plainWrapException(Unknown
Source)
at
org.apache.derby.impl.store.raw.RawStore.backupAndEnableLogArchiveMode(Unknown
Source)
at
org.apache.derby.impl.store.access.RAMAccessManager.backupAndEnableLogArchiveMode(Unknown
Source)
at
org.apache.derby.impl.db.BasicDatabase.backupAndEnableLogArchiveMode(Unknown
Source)
at
org.apache.derby.catalog.SystemProcedures.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE(Unknown
Source)
at
org.apache.derby.exe.ac2a2882b5x015dx2354xca08xffff89e47f210.g0(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at
org.apache.derby.impl.services.reflect.ReflectMethod.invoke(Unknown Source)
at
org.apache.derby.impl.sql.execute.CallStatementResultSet.open(Unknown
Source)
at
org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown
Source)
at
org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
at
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedCallableStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source)
at org.apache.derby.impl.drda.DRDAStatement.execute(Unknown Source)
at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSTT(Unknown
Source)
at
org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.derby.impl.store.raw.data.CachedPage.setPageArray(Unknown Source)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(Unknown Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
at
org.apache.derby.impl.store.raw.data.FileContainer.getLatchedPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer.backupContainer(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.BaseContainerHandle.backupContainer(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.BaseDataFileFactory.backupDataFiles(Unknown
Source)
at org.apache.derby.impl.store.raw.RawStore.backup(Unknown Source)
at org.apache.derby.impl.store.raw.RawStore.backup(Unknown Source)
Regards,
p.
On 7 July 2017 at 04:59, Bryan Pendleton <bp...@gmail.com> wrote:
>
>
> how does derby.storage.pageCacheSize parameter (
>> https://db.apache.org/derby/docs/10.13/ref/rrefproper81359.html) work
>> with database that has multiple page sizes--tables with default 4096 bytes
>> and tables with long/blob of 32768 byte pages?
>>
>>
> Hi Peter,
>
> I'm not 100% sure how this works; I think you should run some experiments.
>
> Here's what I *think* the behavior is:
> 1) The page cache is sized in 4K pages, so if you set pageCacheSize=1024,
> you get 4 meg of page cache memory
> 2) Tables with 4K pages simply use the cache as you expect.
> 3) Tables with 32K pages chew up 8 cache "pages" at a time, each 32K chunk
> of page cache holding 1 32K page from that large-page table.
>
> Some stuff I found while searching around, which might give you some ideas
> for experiments you could run:
>
> http://apache-database.10148.n7.nabble.com/out-of-memory-
> when-writing-blobs-td100948.html
>
> and
>
> https://issues.apache.org/jira/browse/DERBY-4537
>
> Sorry I'm not of much more help here.
>
> bryan
>
>
--
Peter Ondruška
--
kaibo, s.r.o., ID 28435036, registered with the commercial register
administered by the Municipal Court in Prague, section C, insert 141269.
Registered office and postal address: kaibo, s.r.o., Kališnická 379/10,
Prague 3, 130 00, Czech Republic.
https://www.kaibo.eu
Re: Page cache sizing
Posted by Bryan Pendleton <bp...@gmail.com>.
how does derby.storage.pageCacheSize parameter (https://db.apache.org/derby/
> docs/10.13/ref/rrefproper81359.html) work with database that has multiple
> page sizes--tables with default 4096 bytes and tables with long/blob of
> 32768 byte pages?
>
>
Hi Peter,
I'm not 100% sure how this works; I think you should run some experiments.
Here's what I *think* the behavior is:
1) The page cache is sized in 4K pages, so if you set pageCacheSize=1024,
you get 4 meg of page cache memory
2) Tables with 4K pages simply use the cache as you expect.
3) Tables with 32K pages chew up 8 cache "pages" at a time, each 32K chunk
of page cache holding 1 32K page from that large-page table.
Some stuff I found while searching around, which might give you some ideas
for experiments you could run:
http://apache-database.10148.n7.nabble.com/out-of-memory-when-writing-blobs-td100948.html
and
https://issues.apache.org/jira/browse/DERBY-4537
Sorry I'm not of much more help here.
bryan