You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by David Van Couvering <da...@vancouvering.com> on 2010/08/23 19:49:36 UTC
Fluctuating disk overhead
Hi, all. I have a test that updates 100 rows, each of which contains an int
primary key and a 300-byte BLOB:
--
David W. Van Couvering
http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering
Re: Fluctuating disk overhead
Posted by David Van Couvering <da...@vancouvering.com>.
Thanks, Knut! It may be fine to stick with the defaults - I am
contemplating whether this is all that relevant for our customers, versus
just something that our QA team found working with smaller data sets...
On Tue, Aug 24, 2010 at 7:32 AM, Knut Anders Hatlen
<kn...@oracle.com>wrote:
> David Van Couvering <da...@vancouvering.com> writes:
>
> > Can you configure checkpoint frequency, or the maximum size of the
> > log?
>
> Yes, but it's not documented. These properties may be useful:
>
> derby.storage.logSwitchInterval - how often to switch to a new log file
> (number of bytes, default 1MB, min 100000 bytes, max 128 MB).
>
> derby.storage.checkpointInterval - attempt a checkpoint when this amount
> of log in bytes is reached (default 10MB, max 128MB, min 100000 bytes).
>
> --
> Knut Anders
>
--
David W. Van Couvering
http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering
Re: Fluctuating disk overhead
Posted by Knut Anders Hatlen <kn...@oracle.com>.
David Van Couvering <da...@vancouvering.com> writes:
> Can you configure checkpoint frequency, or the maximum size of the
> log?
Yes, but it's not documented. These properties may be useful:
derby.storage.logSwitchInterval - how often to switch to a new log file
(number of bytes, default 1MB, min 100000 bytes, max 128 MB).
derby.storage.checkpointInterval - attempt a checkpoint when this amount
of log in bytes is reached (default 10MB, max 128MB, min 100000 bytes).
--
Knut Anders
Re: Fluctuating disk overhead
Posted by David Van Couvering <da...@vancouvering.com>.
Yea, that makes sense. If this were the case, would forcing checkpoints
help the situation? Can you configure checkpoint frequency, or the maximum
size of the log?
Thanks,
David
On Mon, Aug 23, 2010 at 4:17 PM, Mike Matrigali <mi...@sbcglobal.net>wrote:
> David Van Couvering wrote:
>
> It will probably be a lot easier to explain if you du
> the "log" directory and the "seg0" directory separately
> each time. And for more detail do a ls -l of the "log"
> directory. My guess is that it is due to logging, and
> incremental log file cleanup, but would need more data
> to confirm.
>
> Sorry, about that, new mouse and I did a premature send...
>>
>> As I was saying...
>>
>> Here is the table definition:
>>
>> CREATE TABLE BLOBTEST(ID INTEGER PRIMARY KEY, BLOBCOL BLOB)
>>
>> I insert 100 rows, and then do a loop updating the 100 rows over and over
>> again.
>>
>> When I watch the disk usage of the database directory, I see this pattern
>> repeated over and over again.
>>
>> I tried adding a call to compress the table after every 100 updates, but
>> this had no impact.
>>
>> Can you explain what is going on and what, if anything, I can do to keep
>> the size small?
>>
>> Thanks,
>>
>> David
>>
>> $ du -s -h BLOBTEST
>> 3.3M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 3.4M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 4.9M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 5.9M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 6.5M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 7.0M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 8.6M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 8.7M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 8.7M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 9.7M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 11M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 12M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 11M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 13M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 14M BLOBTEST
>>
>> $ du -s -h BLOBTEST
>> 4.3M BLOBTEST
>>
>> On Mon, Aug 23, 2010 at 10:49 AM, David Van Couvering <
>> david@vancouvering.com <ma...@vancouvering.com>> wrote:
>>
>> Hi, all. I have a test that updates 100 rows, each of which
>> contains an int primary key and a 300-byte BLOB:
>>
>>
>> -- David W. Van Couvering
>>
>> http://www.linkedin.com/in/davidvc
>> http://davidvancouvering.blogspot.com
>> http://twitter.com/dcouvering
>>
>>
>>
>>
>> --
>> David W. Van Couvering
>>
>> http://www.linkedin.com/in/davidvc
>> http://davidvancouvering.blogspot.com
>> http://twitter.com/dcouvering
>>
>
>
--
David W. Van Couvering
http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering
Re: Fluctuating disk overhead
Posted by Mike Matrigali <mi...@sbcglobal.net>.
David Van Couvering wrote:
It will probably be a lot easier to explain if you du
the "log" directory and the "seg0" directory separately
each time. And for more detail do a ls -l of the "log"
directory. My guess is that it is due to logging, and
incremental log file cleanup, but would need more data
to confirm.
> Sorry, about that, new mouse and I did a premature send...
>
> As I was saying...
>
> Here is the table definition:
>
> CREATE TABLE BLOBTEST(ID INTEGER PRIMARY KEY, BLOBCOL BLOB)
>
> I insert 100 rows, and then do a loop updating the 100 rows over and
> over again.
>
> When I watch the disk usage of the database directory, I see this
> pattern repeated over and over again.
>
> I tried adding a call to compress the table after every 100 updates, but
> this had no impact.
>
> Can you explain what is going on and what, if anything, I can do to keep
> the size small?
>
> Thanks,
>
> David
>
> $ du -s -h BLOBTEST
> 3.3M BLOBTEST
>
> $ du -s -h BLOBTEST
> 3.4M BLOBTEST
>
> $ du -s -h BLOBTEST
> 4.9M BLOBTEST
>
> $ du -s -h BLOBTEST
> 5.9M BLOBTEST
>
> $ du -s -h BLOBTEST
> 6.5M BLOBTEST
>
> $ du -s -h BLOBTEST
> 7.0M BLOBTEST
>
> $ du -s -h BLOBTEST
> 8.6M BLOBTEST
>
> $ du -s -h BLOBTEST
> 8.7M BLOBTEST
>
> $ du -s -h BLOBTEST
> 8.7M BLOBTEST
>
> $ du -s -h BLOBTEST
> 9.7M BLOBTEST
>
> $ du -s -h BLOBTEST
> 11M BLOBTEST
>
> $ du -s -h BLOBTEST
> 12M BLOBTEST
>
> $ du -s -h BLOBTEST
> 11M BLOBTEST
>
> $ du -s -h BLOBTEST
> 13M BLOBTEST
>
> $ du -s -h BLOBTEST
> 14M BLOBTEST
>
> $ du -s -h BLOBTEST
> 4.3M BLOBTEST
>
> On Mon, Aug 23, 2010 at 10:49 AM, David Van Couvering
> <david@vancouvering.com <ma...@vancouvering.com>> wrote:
>
> Hi, all. I have a test that updates 100 rows, each of which
> contains an int primary key and a 300-byte BLOB:
>
>
>
> --
> David W. Van Couvering
>
> http://www.linkedin.com/in/davidvc
> http://davidvancouvering.blogspot.com
> http://twitter.com/dcouvering
>
>
>
>
> --
> David W. Van Couvering
>
> http://www.linkedin.com/in/davidvc
> http://davidvancouvering.blogspot.com
> http://twitter.com/dcouvering
Re: Fluctuating disk overhead
Posted by David Van Couvering <da...@vancouvering.com>.
Sorry, about that, new mouse and I did a premature send...
As I was saying...
Here is the table definition:
CREATE TABLE BLOBTEST(ID INTEGER PRIMARY KEY, BLOBCOL BLOB)
I insert 100 rows, and then do a loop updating the 100 rows over and over
again.
When I watch the disk usage of the database directory, I see this pattern
repeated over and over again.
I tried adding a call to compress the table after every 100 updates, but
this had no impact.
Can you explain what is going on and what, if anything, I can do to keep the
size small?
Thanks,
David
$ du -s -h BLOBTEST
3.3M BLOBTEST
$ du -s -h BLOBTEST
3.4M BLOBTEST
$ du -s -h BLOBTEST
4.9M BLOBTEST
$ du -s -h BLOBTEST
5.9M BLOBTEST
$ du -s -h BLOBTEST
6.5M BLOBTEST
$ du -s -h BLOBTEST
7.0M BLOBTEST
$ du -s -h BLOBTEST
8.6M BLOBTEST
$ du -s -h BLOBTEST
8.7M BLOBTEST
$ du -s -h BLOBTEST
8.7M BLOBTEST
$ du -s -h BLOBTEST
9.7M BLOBTEST
$ du -s -h BLOBTEST
11M BLOBTEST
$ du -s -h BLOBTEST
12M BLOBTEST
$ du -s -h BLOBTEST
11M BLOBTEST
$ du -s -h BLOBTEST
13M BLOBTEST
$ du -s -h BLOBTEST
14M BLOBTEST
$ du -s -h BLOBTEST
4.3M BLOBTEST
On Mon, Aug 23, 2010 at 10:49 AM, David Van Couvering <
david@vancouvering.com> wrote:
> Hi, all. I have a test that updates 100 rows, each of which contains an
> int primary key and a 300-byte BLOB:
>
>
>
> --
> David W. Van Couvering
>
> http://www.linkedin.com/in/davidvc
> http://davidvancouvering.blogspot.com
> http://twitter.com/dcouvering
>
--
David W. Van Couvering
http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering