pagesize/pagecount change in beta4

[Catalin Braica <ca...@museglobal.ro>]

Hello

I have a database that uses a lot of collections (several thousand) and 
wanted to move from xindice 1.1 beta2 to 1.1 beta4, and found the 
following problem: http://issues.apache.org/bugzilla/show_bug.cgi?id=31159

Looking at the code (Paged.java) I see that in both beta2 and beta4 the 
pagesize was 4096, and pagecount was 1024, but looking at the .tbl files 
on the disk (just a 'blank' database with system collection) shows great 
difference in size between beta2 (12Kb per .tbl) and beta4 (6Mb per 
.tbl). So what really changed ?

Having a database with many collections upgrading to beta4 means a lot 
of space.
So what can I do ?

Looking at the code, I can edit Paged.java and changed the pagesize and 
pagecount to match my needs. Does anyone have any suggestion on finding 
the right ones (I assume changing these can impact performance) ? Have 
anyone played with these ?

And moreover, is there a way so I can change through the configuration 
file the pagesize for all the collections created (some of their names 
are IDs computed at runtime)? I did not succeeded  in setting pagesize 
in the configuration file not even per collection name, for a single 
collection.
Also since collections in my case are created through XML RPC, can I 
supply a pagesize option when creating a collection ? (as an alternate 
method, relying on the client instead of server).

Thanks
Catalin Braica

Re: pagesize/pagecount change in beta4

[Vadim Gritsenko <va...@reverycodes.com>]

Catalin Braica wrote:
> I changed the pagesize/pagecount in the code (if everything turns well 
> I'll make them configurable) and I am now testing what are the 

Hi Catalin,

I want to let you know that you can configure both pageSize and pageCount now in 
latest SVN code. It still has one minor issue in filer which I'm going to fix 
before next release, but at least you can now configure the file size for your 
collections.


> implications of having 20-60k collections (memory/descriptors/disk space 
> usage). Up to now it didn't turned out too well (I'll write a summary 
> once I finish and reach some conclusions). I might need to play with the 
> number of descriptors / collection as well.

Having 20-60k collections does not sound right to me... To make an analogy, it 
is if somebody would install Oracle and created 20-60k tablespaces in there...

Still, I'd like to hear about your results :)

Vadim

Re: pagesize/pagecount change in beta4

[Catalin Braica <ca...@museglobal.ro>]

Hello

Thanks for the reply

I changed the pagesize/pagecount in the code (if everything turns well 
I'll make them configurable) and I am now testing what are the 
implications of having 20-60k collections (memory/descriptors/disk space 
usage). Up to now it didn't turned out too well (I'll write a summary 
once I finish and reach some conclusions). I might need to play with the 
number of descriptors / collection as well.

Vadim Gritsenko wrote:
> Catalin Braica wrote:
>> I have a database that uses a lot of collections (several thousand) 
>> and wanted to move from xindice 1.1 beta2 to 1.1 beta4, and found the 
>> following problem: 
>> http://issues.apache.org/bugzilla/show_bug.cgi?id=31159
>>
>> Looking at the code (Paged.java) I see that in both beta2 and beta4 
>> the pagesize was 4096, and pagecount was 1024, but looking at the 
>> .tbl files on the disk (just a 'blank' database with system 
>> collection) shows great difference in size between beta2 (12Kb per 
>> .tbl) and beta4 (6Mb per .tbl). So what really changed ?
>
> beta2 did not allocate file properly, and beta4 also has an error: it 
> should be 4Mb, not 6Mb. Current SVN has the fix.
>
>> Having a database with many collections upgrading to beta4 means a 
>> lot of space. So what can I do ?
>
> I'll be adding configurability for pagecount parameter (IIUC pagesize 
> is already configurable) sometime soon. You can try doing it yourself 
> if you need it sooner.
>
>
>> Looking at the code, I can edit Paged.java and changed the pagesize 
>> and pagecount to match my needs. Does anyone have any suggestion on 
>> finding the right ones (I assume changing these can impact 
>> performance) ? Have anyone played with these ?
>
> Reduce pagecount to whatever you feel comfortable with. Do not reduce 
> pagesize too much.
>
>
>> And moreover, is there a way so I can change through the 
>> configuration file the pagesize for all the collections created (some 
>> of their names are IDs computed at runtime)?
>
> This feature is planned.
>
>
>> I did not succeeded  in setting pagesize in the configuration file 
>> not even per collection name, for a single collection.
>> Also since collections in my case are created through XML RPC, can I 
>> supply a pagesize option when creating a collection ? (as an 
>> alternate method, relying on the client instead of server).
>
> Yes. You can pass collection config to the CreateCollection method. 
> See CreateCollection.java.
>
> If your client is in java see 
> org.apache.xindice.integration.client.XmlDbClient for sample code.
>
> Vadim
>

Re: pagesize/pagecount change in beta4

[Vadim Gritsenko <va...@reverycodes.com>]

Catalin Braica wrote:
> I have a database that uses a lot of collections (several thousand) and 
> wanted to move from xindice 1.1 beta2 to 1.1 beta4, and found the 
> following problem: http://issues.apache.org/bugzilla/show_bug.cgi?id=31159
> 
> Looking at the code (Paged.java) I see that in both beta2 and beta4 the 
> pagesize was 4096, and pagecount was 1024, but looking at the .tbl files 
> on the disk (just a 'blank' database with system collection) shows great 
> difference in size between beta2 (12Kb per .tbl) and beta4 (6Mb per 
> .tbl). So what really changed ?

beta2 did not allocate file properly, and beta4 also has an error: it should be 
4Mb, not 6Mb. Current SVN has the fix.

> Having a database with many collections upgrading to beta4 means a lot 
> of space. So what can I do ?

I'll be adding configurability for pagecount parameter (IIUC pagesize is already 
configurable) sometime soon. You can try doing it yourself if you need it sooner.


> Looking at the code, I can edit Paged.java and changed the pagesize and 
> pagecount to match my needs. Does anyone have any suggestion on finding 
> the right ones (I assume changing these can impact performance) ? Have 
> anyone played with these ?

Reduce pagecount to whatever you feel comfortable with. Do not reduce pagesize 
too much.


> And moreover, is there a way so I can change through the configuration 
> file the pagesize for all the collections created (some of their names 
> are IDs computed at runtime)?

This feature is planned.


> I did not succeeded  in setting pagesize 
> in the configuration file not even per collection name, for a single 
> collection.
> Also since collections in my case are created through XML RPC, can I 
> supply a pagesize option when creating a collection ? (as an alternate 
> method, relying on the client instead of server).

Yes. You can pass collection config to the CreateCollection method. See 
CreateCollection.java.

If your client is in java see org.apache.xindice.integration.client.XmlDbClient 
for sample code.

Vadim