You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Brian Peterson <di...@verizon.net> on 2009/03/03 00:27:40 UTC
initial page allocation versus incremental allocation
I see that there's a property to allow configuring the number of pages to
initially allocate to a table, derby.storage.initialPages, but there isn't a
property to allow for setting the number of pages to allocate when
incrementally expanding the file container. It looks like RawStoreFactory
might've allowed for this with
public static final String PRE_ALLOCATE_PAGE =
"derby.storage.pagePerAllocation";
but this isn't reference by anything I can find. FileContainer fixes the
incremental expansion to 8 pages with the DEFAULT_PRE_ALLOC_SIZE constant.
What was the reason for not allowing the pre-allocation setting to be
configurable? Were there adverse affects on FileContainer if it was
increased to something like 100 pages?
Brian
Re: initial page allocation versus incremental allocation
Posted by Bryan Pendleton <bp...@amberpoint.com>.
> idea, if they were used at all was for the software to automatically
> determine what the best settings and up them dynamically rather than
> force/allow user to do so.
Another idea would be to try to do something adaptive, along the lines of:
- if the table is currently smaller than N pages, use an incremental
allocation of 8 pages
- if the table is larger than N pages, but smaller than 2*N pages,
use an incremental allocation of 24 pages
- if the table is larger than 2*N pages, but smaller than 10*N pages,
use an incremental allocation of 128 pages
- if the table is larger than 10*N pages, use an incremental
allocation of 256 pages
Or something like that.
I guess I'm just trying to suggest that it might be possible to be
a zero-admin embedded DB and still scale well at page allocation for
very large tables.
This discussion is probably best suited for derby-dev, and as Knut said
it would be nice if there was a JIRA that tracked this for the future.
thanks,
bryan
Re: initial page allocation versus incremental allocation
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Whoever decides to work on this should do more than just document, as
these params are not tested.
They were added in the past for ad-hoc testing and performance
debugging. As has been suggested they were never fully
implemented/documented as they did not fit in with the original goal
of the product - ie. be a zero-admin embedded db for smaller devices. The
idea, if they were used at all was for the software to automatically
determine what the best settings and up them dynamically rather than
force/allow user to do so.
As I remember the defaults have something to do with the following:
o Start with no preallocation as originally users actually would notice
if a 2k page table started out with 16k vs 2k for an empty table.
o default to 8 as again the "standard" user was space limited and did
not want to possibly waste space.
o 1000 is just a round number, but if it actually gets documented I
would see if the code can actually support a number that is more than
the number of pages that can be tracked by a single allocation page (or
maybe 2). This number is dependent on page sizes, so worst case would
be allocation on a 2k page table. Should just add some testing if it is
to be supported.
Knut Anders Hatlen wrote:
> Brian Peterson <pu...@verizon.net> writes:
>
>> Won't this still limit the setting to the MAX_PRE_ALLOC_SIZE? Why
>> have the maximum setting? Or, at least, why limit it to just 8 pages?
>
> MAX_PRE_ALLOC_SIZE is 1000, it's the default that is just 8 pages, as
> far as I can see. (And no, I don't know why it is limited to 1000 pages,
> but I wouldn't expect many objections if a request to increase the limit
> was filed in JIRA - https://issues.apache.org/jira/browse/DERBY)
>
Re: initial page allocation versus incremental allocation
Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Brian Peterson <pu...@verizon.net> writes:
> Won't this still limit the setting to the MAX_PRE_ALLOC_SIZE? Why
> have the maximum setting? Or, at least, why limit it to just 8 pages?
MAX_PRE_ALLOC_SIZE is 1000, it's the default that is just 8 pages, as
far as I can see. (And no, I don't know why it is limited to 1000 pages,
but I wouldn't expect many objections if a request to increase the limit
was filed in JIRA - https://issues.apache.org/jira/browse/DERBY)
--
Knut Anders
RE: initial page allocation versus incremental allocation
Posted by Brian Peterson <pu...@verizon.net>.
Won't this still limit the setting to the MAX_PRE_ALLOC_SIZE? Why have the maximum setting? Or, at least, why limit it to just 8 pages? If it is configurable, then applications that need the option have it available, and those that don't can continue using the default settings. I'm surprised there's an option to configure the initial allocation but not the increment size.
What I've been trying to do is use Derby in a client application that needs to download more data than will fit into heap. The objects get serialized into rows and are indexed by the row number and an additional java.util.UUID that identifies the object. The client gui can analyze the data and display the results and the raw data as required.
The loading works great up to the initial allocation of pages, then it crawls, which cascades through and disrupts the whole application. It looks like I could solve this by configuring the page-increment size just like the initial allocation setting.
I had hoped to use Derby as a kind of general purpose heap management solution. It has been doing a great job of limiting the amount of heap it uses, and it integrates well with the rest of the application, but the slowdown of the table loading is killing me.
Brian
-----Original Message-----
From: Knut.Hatlen@Sun.COM [mailto:Knut.Hatlen@Sun.COM]
Sent: Tuesday, March 03, 2009 3:28 AM
To: Derby Discussion
Subject: Re: initial page allocation versus incremental allocation
Brian Peterson <di...@verizon.net> writes:
> I see that there’s a property to allow configuring the number of pages
> to initially allocate to a table, derby.storage.initialPages, but
> there isn’t a property to allow for setting the number of pages to
> allocate when incrementally expanding the file container. It looks
> like RawStoreFactory might’ve allowed for this with
>
> public static final String PRE_ALLOCATE_PAGE = “derby.storage.pagePerAllocation”;
>
> but this isn’t reference by anything I can find.
I haven't tested that it actually works, but it appears to be referenced
in FileContainer.createInfoFromProp():
PreAllocSize =
PropertyUtil.getServiceInt(tc, createArgs,
RawStoreFactory.PRE_ALLOCATE_PAGE,
MIN_PRE_ALLOC_SIZE,
MAX_PRE_ALLOC_SIZE,
DEFAULT_PRE_ALLOC_SIZE /* default */);
If it turns out that setting the property works, we should probably try
to get it into the documentation, as it looks like it could be useful.
--
Knut Anders
Re: initial page allocation versus incremental allocation
Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Brian Peterson <di...@verizon.net> writes:
> I see that there’s a property to allow configuring the number of pages
> to initially allocate to a table, derby.storage.initialPages, but
> there isn’t a property to allow for setting the number of pages to
> allocate when incrementally expanding the file container. It looks
> like RawStoreFactory might’ve allowed for this with
>
> public static final String PRE_ALLOCATE_PAGE = “derby.storage.pagePerAllocation”;
>
> but this isn’t reference by anything I can find.
I haven't tested that it actually works, but it appears to be referenced
in FileContainer.createInfoFromProp():
PreAllocSize =
PropertyUtil.getServiceInt(tc, createArgs,
RawStoreFactory.PRE_ALLOCATE_PAGE,
MIN_PRE_ALLOC_SIZE,
MAX_PRE_ALLOC_SIZE,
DEFAULT_PRE_ALLOC_SIZE /* default */);
If it turns out that setting the property works, we should probably try
to get it into the documentation, as it looks like it could be useful.
--
Knut Anders
RE: initial page allocation versus incremental allocation
Posted by de...@segel.com.
I think that the simple answer is that Cloudscape wasn't designed as a
general purpose relational database. Its target niche was embedded and
smaller apps.
So it sounds like the issue you were having is solved. That is, once you hit
your preset size, the default allocations is too small and its impacting
your ability to increase your table space.
It sounds like an enhancement to me.
BTW, if you want to 'hack' this to test the theory out, just change the
DEFAULT_PRE_ALLOC_SIZE constant in the File Container to be something like
1000.
Then run your test again.
_____
From: Brian Peterson [mailto:dianeayers@verizon.net]
Sent: Monday, March 02, 2009 5:28 PM
To: 'Derby Discussion'
Subject: initial page allocation versus incremental allocation
I see that there's a property to allow configuring the number of pages to
initially allocate to a table, derby.storage.initialPages, but there isn't a
property to allow for setting the number of pages to allocate when
incrementally expanding the file container. It looks like RawStoreFactory
might've allowed for this with
public static final String PRE_ALLOCATE_PAGE =
"derby.storage.pagePerAllocation";
but this isn't reference by anything I can find. FileContainer fixes the
incremental expansion to 8 pages with the DEFAULT_PRE_ALLOC_SIZE constant.
What was the reason for not allowing the pre-allocation setting to be
configurable? Were there adverse affects on FileContainer if it was
increased to something like 100 pages?
Brian