Re: more background threads

[Mike Matrigali <mi...@sbcglobal.net>]

I have changed the subject, as I completely missed the original post
which had something to do with adding Junit tests.

I am not sure what is the right solution here, but getting a discussion
going would be good.

Currently a number of store actions are queued in "post commit" mode,
which means they should be executed until after the transaction which
queued them commits.  Currently there is one background thread which
processes these, if it gets too full then the work is done by the actual
thread which queued the work.   Most of the post commit work involves
claiming space from deleted rows after their transaction commits.

Going forward there is going to be a need for more background work.  I
soon will be posting the first phase of work to allow for returning
space back to the operating system, eventually it would be best if this
work was also done in background, somehow automatically queued by the
system.

I would also recommend coming up with a usage scenario which shows a
problem before coding up a solution.  I believe a test with lots of
users doing insert and delete should eventually show the background task
being bogged down -- but I am not sure if moving work to additional
threads is much better than just spreading the work out across the
existing user threads.

The code for the current background thread can be found in:
opensource/java/engine/org/apache/derby/impl/services/daemon

An example of one of the unit of work put on the queue is in:
opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java

Dan is probably the person who most recently worked on this code, and
should have some comments in this area.  He should be back active on the
list early next week.

Note another interesting area of research/coding would be to see how
derby scales on larger number of processor machines.  Not much work has
been done at all on machines with more than 2 processors.  The system
has been designed from bottom up to be multi-threaded, but not much
testing/monitoring has been done on 4 or more processor machines.   The
following single threading points exist in derby:
    o each user query is executed by a single thread.
    o the locking system in protected by a single java synchonization point.
    o copying log records into the log is a single sync point
    o finding a buffer in the buffer cache is a single sync point

All of these seemed to be reasonable designs for 1, 2 and 4 way machines.

/mikem


David Van Couvering wrote:

> I noticed on the todo list there is a need to have more than one
> background thread to enable better scalability with lots of client
> connections.  I'm trying to find a way to gently work my way into doing
> some work on Derby, and this seemed like a project of small enough scope
> to get my feet wet.  Is there any background on this, or should I just
> jump right in?  I didn't see any discussion of this on the list...
> 
> Thanks,
> 
> David
> 

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

OK, that worked, I was able to step into Derby code, thanks!

David

Daniel John Debrunner wrote:

> David Van Couvering wrote:
> 
> 
>>OK...  I'm still trying to get clear on your concepts and terms -- how
>>is a "system" defined?  I'm thinking "all of Derby" within a given VM?
> 
> 
> Correct. A system corresponds to the loading of Derby's JDBC driver
> within a JVM, a system can have multiple databases open within that JVM.
> 
> 
> 
>>Before I spend a lot more time on this: are there other things that are
>>reasonable in scale and scope you could really use my help on that are
>>generally felt to be of higher priority?  I don't need to spend a lot of
>>time on this if it currently works pretty well and is not causing a lot
>>of pain or problems...
> 
> 
> Apart from fixing the issue of too many Java threads when a large number
> of databases are active, I see the cleanup in this area as a step
> towards supporting:
> 
>  - timed events - java.util.TimerTask integration, for setQueryTimeout
> at least
> 
>  - Serviceable items that can be executed during idle time, e.g.
> background data flushing, space reclaimation, background statistics
> gathering, etc.
> 
> While these items could be fitted into the current scheme, I think a
> cleanup would make it easier.
> 
> Dan.
> 

Re: more background threads

[Daniel John Debrunner <dj...@debrunners.com>]

David Van Couvering wrote:

> So I suppose conceivably you could have multiple systems in a VM if you
> have multiple classloaders, each of which loads its own copy of the JDBC
> driver class...

Yes, I thought about adding that technicality, but in general there will
be one Derby system per vm. So I left it simple. :-)

Dan

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

So I suppose conceivably you could have multiple systems in a VM if you 
have multiple classloaders, each of which loads its own copy of the JDBC 
driver class...

David

Daniel John Debrunner wrote:

> David Van Couvering wrote:
> 
> 
>>OK...  I'm still trying to get clear on your concepts and terms -- how
>>is a "system" defined?  I'm thinking "all of Derby" within a given VM?
> 
> 
> Correct. A system corresponds to the loading of Derby's JDBC driver
> within a JVM, a system can have multiple databases open within that JVM.
> 
> 
> 
>>Before I spend a lot more time on this: are there other things that are
>>reasonable in scale and scope you could really use my help on that are
>>generally felt to be of higher priority?  I don't need to spend a lot of
>>time on this if it currently works pretty well and is not causing a lot
>>of pain or problems...
> 
> 
> Apart from fixing the issue of too many Java threads when a large number
> of databases are active, I see the cleanup in this area as a step
> towards supporting:
> 
>  - timed events - java.util.TimerTask integration, for setQueryTimeout
> at least
> 
>  - Serviceable items that can be executed during idle time, e.g.
> background data flushing, space reclaimation, background statistics
> gathering, etc.
> 
> While these items could be fitted into the current scheme, I think a
> cleanup would make it easier.
> 
> Dan.
> 

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

OK, thanks, sounds good, I'll look further into it.  I'll be in touch...

David

Daniel John Debrunner wrote:

> David Van Couvering wrote:
> 
> 
>>OK...  I'm still trying to get clear on your concepts and terms -- how
>>is a "system" defined?  I'm thinking "all of Derby" within a given VM?
> 
> 
> Correct. A system corresponds to the loading of Derby's JDBC driver
> within a JVM, a system can have multiple databases open within that JVM.
> 
> 
> 
>>Before I spend a lot more time on this: are there other things that are
>>reasonable in scale and scope you could really use my help on that are
>>generally felt to be of higher priority?  I don't need to spend a lot of
>>time on this if it currently works pretty well and is not causing a lot
>>of pain or problems...
> 
> 
> Apart from fixing the issue of too many Java threads when a large number
> of databases are active, I see the cleanup in this area as a step
> towards supporting:
> 
>  - timed events - java.util.TimerTask integration, for setQueryTimeout
> at least
> 
>  - Serviceable items that can be executed during idle time, e.g.
> background data flushing, space reclaimation, background statistics
> gathering, etc.
> 
> While these items could be fitted into the current scheme, I think a
> cleanup would make it easier.
> 
> Dan.
> 

Re: more background threads

[Daniel John Debrunner <dj...@debrunners.com>]

David Van Couvering wrote:

> OK...  I'm still trying to get clear on your concepts and terms -- how
> is a "system" defined?  I'm thinking "all of Derby" within a given VM?

Correct. A system corresponds to the loading of Derby's JDBC driver
within a JVM, a system can have multiple databases open within that JVM.


> Before I spend a lot more time on this: are there other things that are
> reasonable in scale and scope you could really use my help on that are
> generally felt to be of higher priority?  I don't need to spend a lot of
> time on this if it currently works pretty well and is not causing a lot
> of pain or problems...

Apart from fixing the issue of too many Java threads when a large number
of databases are active, I see the cleanup in this area as a step
towards supporting:

 - timed events - java.util.TimerTask integration, for setQueryTimeout
at least

 - Serviceable items that can be executed during idle time, e.g.
background data flushing, space reclaimation, background statistics
gathering, etc.

While these items could be fitted into the current scheme, I think a
cleanup would make it easier.

Dan.

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

Daniel John Debrunner wrote:
> David Van Couvering wrote:
> 
> 
>>OK, I think I'm beginning to get my head around the background thread
>>design.  It seems to be used in a lightweight way by a number of various
>>components, but is most heavily used by the transaction manager to
>>handle post-commit work.
>>As you mentioned, the current design spreads the work across the user
>>threads if the background thread gets backed up, and I agree that
>>overall throughput may not be much improved by having multiple
>>background threads do the work. 
> 
> 
> The intention of the item on the to-do list was to reduce the number of
> threads not increase them :-). The issue is that if a Derby system is
> booted with 1,000 databases, then 1,000 backgound threads are created,
> all just to handle background work. The to-list item was intended change
> the DaemonFactory from being booted per-database to being booted
> per-system.  Thus a single backgound thread would handle the idle
> requests. Then, I agree, the DaemonFactory could be enhanced to support
> a handful of worker threads behind the interface, without the knowledge
> of the caller.

Yes, when I re-read the todo item this is what you are getting at.  I 
can see value in having a pool of threads handle background work, but 
you are right it would be an explosion of threads if you have a scenario 
where a single JVM is handling thousands or even hundreds of databases 
(say one per application in a large application server deployment).  I 
had been unconsciously thinking from the perspective that you had only 
one background thread per VM, not per database.  I agree it really 
doesn't make sense to add worker threads unless they are shared across 
databases.

This implies that whatever test I write to check the behavior out should 
probably mimic a multi-database environment, where connections are 
spread out across databases.  I could set up a mechanism where 
connections are made in a round-robin or random fashion across a 
configurable number of databases...  Does this make sense?

> 
> The work involved I think is more involved than just changing how the
> factory is booted, since at the moment the context is tied to the
> thread, the change would require the context be separated from the
> thread and set dynamically as work came in from each database.
> In the end there would be a DaemonFactory per system and a DaemonService
> per database.
>

OK...  I'm still trying to get clear on your concepts and terms -- how 
is a "system" defined?  I'm thinking "all of Derby" within a given VM?


> 
> 
>>While reading the code, I did encounter one thing that had me
>>concerned.  What I am calling the transaction manager
>>(impl.store.Xact.java) seems to me to be overstepping its
>>responsibilities:  it takes care of running work items when the work
>>item requires immediate attention or if the daemon thread is backed up. 
>>This logic is not made available to any of the other components that
>>make use of the daemon service.  One can imagine this logic being
>>refactored out and made generally available to all clients of the daemon
>>service.  Is there a specific reason for having this code in Xact.java?
> 
> 
> I think it was a quick change to address a customer specific issue. I
> agree that cleanup is needed here, I once started some code that tried
> to cleanup the Serviceable intefaces, so that there was one set of
> constants (something like DONE,REQUEUE,IMMEDIATE, USERSPACE) and thus
> similar state is not separated over the return values from
> performWork(), serviceASAP and serviceImmediately() and the value passed
> into the enqueue method).

Yes, that was another odd thing I noticed and decided I would get back 
to later to try to understand.  The exact semantics of what happens when 
with the current implementation is a bit complex...  I could see this 
project as an opportunity to try and simplify some of this as you have 
discussed.

> Currently it is not well defined what should happen if the serviceNow
> parameter passed to enqueue is in conflict with the returns from
> serviceNow or serviceImmediately(). You can see that some of the
> implementations of Serviceable implement the current methods
> inconsistently (from memory).
> 

I hadn't noticed that yet, but I would not be surprised...

> 
> Thus I imagined a single method on Serviceable that indicated how the
> operation wanted to be serviced, make performWork void and don't have a
> serviceNow on enqueue.
> 

Yes, that does sound nice.

Before I spend a lot more time on this: are there other things that are 
reasonable in scale and scope you could really use my help on that are 
generally felt to be of higher priority?  I don't need to spend a lot of 
time on this if it currently works pretty well and is not causing a lot 
of pain or problems...

Thanks,

David

> 
> Dan.
> 

Re: more background threads

[Daniel John Debrunner <dj...@debrunners.com>]

David Van Couvering wrote:

> OK, I think I'm beginning to get my head around the background thread
> design.  It seems to be used in a lightweight way by a number of various
> components, but is most heavily used by the transaction manager to
> handle post-commit work.
> As you mentioned, the current design spreads the work across the user
> threads if the background thread gets backed up, and I agree that
> overall throughput may not be much improved by having multiple
> background threads do the work. 

The intention of the item on the to-do list was to reduce the number of
threads not increase them :-). The issue is that if a Derby system is
booted with 1,000 databases, then 1,000 backgound threads are created,
all just to handle background work. The to-list item was intended change
the DaemonFactory from being booted per-database to being booted
per-system.  Thus a single backgound thread would handle the idle
requests. Then, I agree, the DaemonFactory could be enhanced to support
a handful of worker threads behind the interface, without the knowledge
of the caller.

The work involved I think is more involved than just changing how the
factory is booted, since at the moment the context is tied to the
thread, the change would require the context be separated from the
thread and set dynamically as work came in from each database.
In the end there would be a DaemonFactory per system and a DaemonService
per database.


> While reading the code, I did encounter one thing that had me
> concerned.  What I am calling the transaction manager
> (impl.store.Xact.java) seems to me to be overstepping its
> responsibilities:  it takes care of running work items when the work
> item requires immediate attention or if the daemon thread is backed up. 
> This logic is not made available to any of the other components that
> make use of the daemon service.  One can imagine this logic being
> refactored out and made generally available to all clients of the daemon
> service.  Is there a specific reason for having this code in Xact.java?

I think it was a quick change to address a customer specific issue. I
agree that cleanup is needed here, I once started some code that tried
to cleanup the Serviceable intefaces, so that there was one set of
constants (something like DONE,REQUEUE,IMMEDIATE, USERSPACE) and thus
similar state is not separated over the return values from
performWork(), serviceASAP and serviceImmediately() and the value passed
into the enqueue method).
Currently it is not well defined what should happen if the serviceNow
parameter passed to enqueue is in conflict with the returns from
serviceNow or serviceImmediately(). You can see that some of the
implementations of Serviceable implement the current methods
inconsistently (from memory).


Thus I imagined a single method on Serviceable that indicated how the
operation wanted to be serviced, make performWork void and don't have a
serviceNow on enqueue.


Dan.

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

OK, I think I'm beginning to get my head around the background thread 
design.  It seems to be used in a lightweight way by a number of various 
components, but is most heavily used by the transaction manager to 
handle post-commit work. 

As you mentioned, the current design spreads the work across the user 
threads if the background thread gets backed up, and I agree that 
overall throughput may not be much improved by having multiple 
background threads do the work.  If the work is coming in too fast, then 
sooner or later you're going to have to throttle back the background 
work and have it handled by the user threads so that it gets done in a 
reasonable amount of time.  However, having multiple background threads 
could help if you have "bursts" of transactions that require post-commit 
work that the daemon threads can "drain out" in the background without 
impacting response time; it seems to me this might be a common 
scenario.  I guess it depends on the application and what you're trying 
to optimize for; if you have something that is constantly inserting and 
deleting records, then it's possible that what exists now is fine.  If 
you have an application that generally doesn't do deletes, but say once 
or twice a day "cleans house", then having more background threads could 
be very useful.

But, as you said, no design should be done until these theories are 
verified by some performance and scalability testing.  I'm thinking of 
having one test that does sort of steady state inserts and deletes, and 
another one that does mostly inserts and updates and then "cleans house" 
when the number of records reaches some maximum.  What do you think?

While reading the code, I did encounter one thing that had me 
concerned.  What I am calling the transaction manager 
(impl.store.Xact.java) seems to me to be overstepping its 
responsibilities:  it takes care of running work items when the work 
item requires immediate attention or if the daemon thread is backed up.  
This logic is not made available to any of the other components that 
make use of the daemon service.  One can imagine this logic being 
refactored out and made generally available to all clients of the daemon 
service.  Is there a specific reason for having this code in Xact.java?

Thanks,

David

Mike Matrigali wrote:

>I have changed the subject, as I completely missed the original post
>which had something to do with adding Junit tests.
>
>I am not sure what is the right solution here, but getting a discussion
>going would be good.
>
>Currently a number of store actions are queued in "post commit" mode,
>which means they should be executed until after the transaction which
>queued them commits.  Currently there is one background thread which
>processes these, if it gets too full then the work is done by the actual
>thread which queued the work.   Most of the post commit work involves
>claiming space from deleted rows after their transaction commits.
>
>Going forward there is going to be a need for more background work.  I
>soon will be posting the first phase of work to allow for returning
>space back to the operating system, eventually it would be best if this
>work was also done in background, somehow automatically queued by the
>system.
>
>I would also recommend coming up with a usage scenario which shows a
>problem before coding up a solution.  I believe a test with lots of
>users doing insert and delete should eventually show the background task
>being bogged down -- but I am not sure if moving work to additional
>threads is much better than just spreading the work out across the
>existing user threads.
>
>The code for the current background thread can be found in:
>opensource/java/engine/org/apache/derby/impl/services/daemon
>
>An example of one of the unit of work put on the queue is in:
>opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java
>
>Dan is probably the person who most recently worked on this code, and
>should have some comments in this area.  He should be back active on the
>list early next week.
>
>Note another interesting area of research/coding would be to see how
>derby scales on larger number of processor machines.  Not much work has
>been done at all on machines with more than 2 processors.  The system
>has been designed from bottom up to be multi-threaded, but not much
>testing/monitoring has been done on 4 or more processor machines.   The
>following single threading points exist in derby:
>    o each user query is executed by a single thread.
>    o the locking system in protected by a single java synchonization point.
>    o copying log records into the log is a single sync point
>    o finding a buffer in the buffer cache is a single sync point
>
>All of these seemed to be reasonable designs for 1, 2 and 4 way machines.
>
>/mikem
>
>
>David Van Couvering wrote:
>
>  
>
>>I noticed on the todo list there is a need to have more than one
>>background thread to enable better scalability with lots of client
>>connections.  I'm trying to find a way to gently work my way into doing
>>some work on Derby, and this seemed like a project of small enough scope
>>to get my feet wet.  Is there any background on this, or should I just
>>jump right in?  I didn't see any discussion of this on the list...
>>
>>Thanks,
>>
>>David
>>
>>    
>>

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

Hi, Mike, I'll look into it.  I'm on vacation most of this week but will 
be back in the saddle next week.

Regarding NIO, the performance improvement was on the network side, 
using the select-like way of handling client requests.  I am not so sure 
of how much better the NIO package is for disk I/O.  I did hear that 
things are significantly better in JDK 1.5, but that's only hearsay at 
this point.

David

Mike Matrigali wrote:

> Any expertise you can lend in the area of I/O or scalability would
> be greatly appreciated.  I only have access to a dual processor
> machine (and it only has 2 700mhz cpu's).  I am only guessing
> currently at where problems might be, if you can identify problems
> then projects should flow from that.
>
> Suresh and I looked at nio for log I/O over a year ago and did not
> see any improvement on intel/windows and/or intel/linux.  I thought
> that we could avoid some OS/JVM buffer copies by using it, but we
> never measured difference.  If you can get log or database I/O to
> go faster using it that would be great.
>
> Note that many standard performance benchmarks will quickly become
> I/O bound on derby, unless the hardware is configured optimally.  Unlike
> most db's derby requires database on a single device and log on a single
> device, assuming that if applications needed better throughput they
> could use the hardware or OS to spread I/O across multiple devices while
> presenting the view of a single device to the JVM and thus to derby.
>
> David Van Couvering wrote:
>
>>
>> Satheesh Bandaram wrote:
>>
>>> Very interesting plans. What kind of application are you thinking of
>>> running on these 4-way or 8 way machines? Will you be using embedded
>>> driver or client-server network driver?
>>>  
>>>
>>
>> We have a few products within Sun who are interested in using Derby 
>> as their embedded data store.  They like its lightweight nature and 
>> the fact that it can be embedded.  Sun of course has higher-scale 
>> machines and customers who may want to scale up, so we want to make 
>> sure Derby can handle that.  I have to check, but I think there is 
>> interest in both the embedded and network drivers.
>>
>>> I am also wondering how is the scalability of Java VMs these days? I
>>> know earlier VMs (a few years back) weren't scaling well beyond 3 or 4
>>> CPUs. Is that any better with Sun or IBM VMs?
>>>  
>>>
>> Well, I don't personally have the details and exact numbers, but I 
>> know a lot of work has been done to improve scalability of the VM.  
>> The latest implementations at Sun use native threads and provide 
>> really good parallel GC implementations.   I know that our app 
>> server, which is written all in Java, scales quite well.  Generally 
>> we don't need to run more than one instance of the app server per 
>> machine on even the much bigger Sun boxes.
>> We also recently converted our web server to start using the new NIO 
>> package and the "select" model for handling incoming connections, and 
>> now our all Java web server scales better than our C-based web 
>> server, which itself has been winning many of the performance 
>> benchmarks out there.  I was actually going to bring this up at some 
>> point as an idea for a TODO at some point -- convert the network IO 
>> and potentially the disk IO subsystems of Derby to start using NIO...
>> Cheers,
>>
>> David
>>
>>> Satheesh
>>>
>>> David Van Couvering wrote:
>>>
>>>  
>>>
>>>> Hi, Mike, thanks for the response and very helpful overview.  At first
>>>> blush it seems like the single daemon could easily be converted to a
>>>> thread pool approach where work is posted to a "dispatcher" who grabs
>>>> a thread and dispatches the work to it.  I say this without having yet
>>>> looked at the code, but in the meantime any reasons why this obviously
>>>> won't work would be much appreciated.
>>>>
>>>> I can work on building up a test case that fills up the background
>>>> thread so we can "prove" that whatever solution we come up with helps
>>>> the system scale better.  I can post a test plan prior to actually
>>>> creating the test to see if you all agree the test looks to be what we
>>>> want it to be.
>>>>
>>>> I can also look into testing Derby scalability on a 4-way or 8-way
>>>> machine, I think some of these are available in our lab.  I would also
>>>> like to do some testing on some of Sun's new multi-core chips, where
>>>> you have 8 threads per core and 4-8 cores per CPU.  Derby seems to be
>>>> well-suited to this architecture but it would be good to see if there
>>>> are any gotchas.  Again, I would proposed these as plans first and get
>>>> your feedback.
>>>>
>>>> What protocol do I use to sort of "identify" this is a sub-project and
>>>> track its progress?  Do I create a JIRA item labelled as an
>>>> "improvement" and assign it to myself?
>>>>
>>>> Thanks,
>>>>
>>>> David
>>>>
>>>> Mike Matrigali wrote:
>>>>
>>>>   
>>>>
>>>>> I have changed the subject, as I completely missed the original post
>>>>> which had something to do with adding Junit tests.
>>>>>
>>>>> I am not sure what is the right solution here, but getting a 
>>>>> discussion
>>>>> going would be good.
>>>>>
>>>>> Currently a number of store actions are queued in "post commit" mode,
>>>>> which means they should be executed until after the transaction which
>>>>> queued them commits.  Currently there is one background thread which
>>>>> processes these, if it gets too full then the work is done by the 
>>>>> actual
>>>>> thread which queued the work.   Most of the post commit work involves
>>>>> claiming space from deleted rows after their transaction commits.
>>>>>
>>>>> Going forward there is going to be a need for more background 
>>>>> work.  I
>>>>> soon will be posting the first phase of work to allow for returning
>>>>> space back to the operating system, eventually it would be best if 
>>>>> this
>>>>> work was also done in background, somehow automatically queued by the
>>>>> system.
>>>>>
>>>>> I would also recommend coming up with a usage scenario which shows a
>>>>> problem before coding up a solution.  I believe a test with lots of
>>>>> users doing insert and delete should eventually show the 
>>>>> background task
>>>>> being bogged down -- but I am not sure if moving work to additional
>>>>> threads is much better than just spreading the work out across the
>>>>> existing user threads.
>>>>>
>>>>> The code for the current background thread can be found in:
>>>>> opensource/java/engine/org/apache/derby/impl/services/daemon
>>>>>
>>>>> An example of one of the unit of work put on the queue is in:
>>>>> opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java 
>>>>>
>>>>>
>>>>>
>>>>> Dan is probably the person who most recently worked on this code, and
>>>>> should have some comments in this area.  He should be back active 
>>>>> on the
>>>>> list early next week.
>>>>>
>>>>> Note another interesting area of research/coding would be to see how
>>>>> derby scales on larger number of processor machines.  Not much 
>>>>> work has
>>>>> been done at all on machines with more than 2 processors.  The system
>>>>> has been designed from bottom up to be multi-threaded, but not much
>>>>> testing/monitoring has been done on 4 or more processor 
>>>>> machines.   The
>>>>> following single threading points exist in derby:
>>>>>   o each user query is executed by a single thread.
>>>>>   o the locking system in protected by a single java synchonization
>>>>> point.
>>>>>   o copying log records into the log is a single sync point
>>>>>   o finding a buffer in the buffer cache is a single sync point
>>>>>
>>>>> All of these seemed to be reasonable designs for 1, 2 and 4 way
>>>>> machines.
>>>>>
>>>>> /mikem
>>>>>
>>>>>
>>>>> David Van Couvering wrote:
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>> I noticed on the todo list there is a need to have more than one
>>>>>> background thread to enable better scalability with lots of client
>>>>>> connections.  I'm trying to find a way to gently work my way into 
>>>>>> doing
>>>>>> some work on Derby, and this seemed like a project of small enough
>>>>>> scope
>>>>>> to get my feet wet.  Is there any background on this, or should I 
>>>>>> just
>>>>>> jump right in?  I didn't see any discussion of this on the list...
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> David
>>>>>>
>>>>>>  
>>>>>>       
>>>>>
>>>>
>>>>   
>>>
>>>
>>>  
>>>
>

Re: more background threads

[Mike Matrigali <mi...@sbcglobal.net>]

Any expertise you can lend in the area of I/O or scalability would
be greatly appreciated.  I only have access to a dual processor
machine (and it only has 2 700mhz cpu's).  I am only guessing
currently at where problems might be, if you can identify problems
then projects should flow from that.

Suresh and I looked at nio for log I/O over a year ago and did not
see any improvement on intel/windows and/or intel/linux.  I thought
that we could avoid some OS/JVM buffer copies by using it, but we
never measured difference.  If you can get log or database I/O to
go faster using it that would be great.

Note that many standard performance benchmarks will quickly become
I/O bound on derby, unless the hardware is configured optimally.  Unlike
most db's derby requires database on a single device and log on a single
device, assuming that if applications needed better throughput they
could use the hardware or OS to spread I/O across multiple devices while
presenting the view of a single device to the JVM and thus to derby.

David Van Couvering wrote:
> 
> Satheesh Bandaram wrote:
> 
>>Very interesting plans. What kind of application are you thinking of
>>running on these 4-way or 8 way machines? Will you be using embedded
>>driver or client-server network driver?
>>  
>>
> 
> We have a few products within Sun who are interested in using Derby as 
> their embedded data store.  They like its lightweight nature and the 
> fact that it can be embedded.  Sun of course has higher-scale machines 
> and customers who may want to scale up, so we want to make sure Derby 
> can handle that.  I have to check, but I think there is interest in both 
> the embedded and network drivers.
> 
>>I am also wondering how is the scalability of Java VMs these days? I
>>know earlier VMs (a few years back) weren't scaling well beyond 3 or 4
>>CPUs. Is that any better with Sun or IBM VMs?
>>  
>>
> Well, I don't personally have the details and exact numbers, but I know 
> a lot of work has been done to improve scalability of the VM.  The 
> latest implementations at Sun use native threads and provide really good 
> parallel GC implementations.   I know that our app server, which is 
> written all in Java, scales quite well.  Generally we don't need to run 
> more than one instance of the app server per machine on even the much 
> bigger Sun boxes. 
> 
> We also recently converted our web server to start using the new NIO 
> package and the "select" model for handling incoming connections, and 
> now our all Java web server scales better than our C-based web server, 
> which itself has been winning many of the performance benchmarks out 
> there.  I was actually going to bring this up at some point as an idea 
> for a TODO at some point -- convert the network IO and potentially the 
> disk IO subsystems of Derby to start using NIO... 
> 
> Cheers,
> 
> David
> 
>>Satheesh
>>
>>David Van Couvering wrote:
>>
>>  
>>
>>>Hi, Mike, thanks for the response and very helpful overview.  At first
>>>blush it seems like the single daemon could easily be converted to a
>>>thread pool approach where work is posted to a "dispatcher" who grabs
>>>a thread and dispatches the work to it.  I say this without having yet
>>>looked at the code, but in the meantime any reasons why this obviously
>>>won't work would be much appreciated.
>>>
>>>I can work on building up a test case that fills up the background
>>>thread so we can "prove" that whatever solution we come up with helps
>>>the system scale better.  I can post a test plan prior to actually
>>>creating the test to see if you all agree the test looks to be what we
>>>want it to be.
>>>
>>>I can also look into testing Derby scalability on a 4-way or 8-way
>>>machine, I think some of these are available in our lab.  I would also
>>>like to do some testing on some of Sun's new multi-core chips, where
>>>you have 8 threads per core and 4-8 cores per CPU.  Derby seems to be
>>>well-suited to this architecture but it would be good to see if there
>>>are any gotchas.  Again, I would proposed these as plans first and get
>>>your feedback.
>>>
>>>What protocol do I use to sort of "identify" this is a sub-project and
>>>track its progress?  Do I create a JIRA item labelled as an
>>>"improvement" and assign it to myself?
>>>
>>>Thanks,
>>>
>>>David
>>>
>>>Mike Matrigali wrote:
>>>
>>>    
>>>
>>>>I have changed the subject, as I completely missed the original post
>>>>which had something to do with adding Junit tests.
>>>>
>>>>I am not sure what is the right solution here, but getting a discussion
>>>>going would be good.
>>>>
>>>>Currently a number of store actions are queued in "post commit" mode,
>>>>which means they should be executed until after the transaction which
>>>>queued them commits.  Currently there is one background thread which
>>>>processes these, if it gets too full then the work is done by the actual
>>>>thread which queued the work.   Most of the post commit work involves
>>>>claiming space from deleted rows after their transaction commits.
>>>>
>>>>Going forward there is going to be a need for more background work.  I
>>>>soon will be posting the first phase of work to allow for returning
>>>>space back to the operating system, eventually it would be best if this
>>>>work was also done in background, somehow automatically queued by the
>>>>system.
>>>>
>>>>I would also recommend coming up with a usage scenario which shows a
>>>>problem before coding up a solution.  I believe a test with lots of
>>>>users doing insert and delete should eventually show the background task
>>>>being bogged down -- but I am not sure if moving work to additional
>>>>threads is much better than just spreading the work out across the
>>>>existing user threads.
>>>>
>>>>The code for the current background thread can be found in:
>>>>opensource/java/engine/org/apache/derby/impl/services/daemon
>>>>
>>>>An example of one of the unit of work put on the queue is in:
>>>>opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java
>>>>
>>>>
>>>>Dan is probably the person who most recently worked on this code, and
>>>>should have some comments in this area.  He should be back active on the
>>>>list early next week.
>>>>
>>>>Note another interesting area of research/coding would be to see how
>>>>derby scales on larger number of processor machines.  Not much work has
>>>>been done at all on machines with more than 2 processors.  The system
>>>>has been designed from bottom up to be multi-threaded, but not much
>>>>testing/monitoring has been done on 4 or more processor machines.   The
>>>>following single threading points exist in derby:
>>>>   o each user query is executed by a single thread.
>>>>   o the locking system in protected by a single java synchonization
>>>>point.
>>>>   o copying log records into the log is a single sync point
>>>>   o finding a buffer in the buffer cache is a single sync point
>>>>
>>>>All of these seemed to be reasonable designs for 1, 2 and 4 way
>>>>machines.
>>>>
>>>>/mikem
>>>>
>>>>
>>>>David Van Couvering wrote:
>>>>
>>>> 
>>>>
>>>>      
>>>>
>>>>>I noticed on the todo list there is a need to have more than one
>>>>>background thread to enable better scalability with lots of client
>>>>>connections.  I'm trying to find a way to gently work my way into doing
>>>>>some work on Derby, and this seemed like a project of small enough
>>>>>scope
>>>>>to get my feet wet.  Is there any background on this, or should I just
>>>>>jump right in?  I didn't see any discussion of this on the list...
>>>>>
>>>>>Thanks,
>>>>>
>>>>>David
>>>>>
>>>>>  
>>>>>        
>>>>>
>>>
>>>    
>>>
>>
>>  
>>

Re: more background threads

[Satheesh Bandaram <sa...@Sourcery.Org>]

Very interesting plans. What kind of application are you thinking of
running on these 4-way or 8 way machines? Will you be using embedded
driver or client-server network driver?

I am also wondering how is the scalability of Java VMs these days? I
know earlier VMs (a few years back) weren't scaling well beyond 3 or 4
CPUs. Is that any better with Sun or IBM VMs?

Satheesh

David Van Couvering wrote:

> Hi, Mike, thanks for the response and very helpful overview.  At first
> blush it seems like the single daemon could easily be converted to a
> thread pool approach where work is posted to a "dispatcher" who grabs
> a thread and dispatches the work to it.  I say this without having yet
> looked at the code, but in the meantime any reasons why this obviously
> won't work would be much appreciated.
>
> I can work on building up a test case that fills up the background
> thread so we can "prove" that whatever solution we come up with helps
> the system scale better.  I can post a test plan prior to actually
> creating the test to see if you all agree the test looks to be what we
> want it to be.
>
> I can also look into testing Derby scalability on a 4-way or 8-way
> machine, I think some of these are available in our lab.  I would also
> like to do some testing on some of Sun's new multi-core chips, where
> you have 8 threads per core and 4-8 cores per CPU.  Derby seems to be
> well-suited to this architecture but it would be good to see if there
> are any gotchas.  Again, I would proposed these as plans first and get
> your feedback.
>
> What protocol do I use to sort of "identify" this is a sub-project and
> track its progress?  Do I create a JIRA item labelled as an
> "improvement" and assign it to myself?
>
> Thanks,
>
> David
>
> Mike Matrigali wrote:
>
>> I have changed the subject, as I completely missed the original post
>> which had something to do with adding Junit tests.
>>
>> I am not sure what is the right solution here, but getting a discussion
>> going would be good.
>>
>> Currently a number of store actions are queued in "post commit" mode,
>> which means they should be executed until after the transaction which
>> queued them commits.  Currently there is one background thread which
>> processes these, if it gets too full then the work is done by the actual
>> thread which queued the work.   Most of the post commit work involves
>> claiming space from deleted rows after their transaction commits.
>>
>> Going forward there is going to be a need for more background work.  I
>> soon will be posting the first phase of work to allow for returning
>> space back to the operating system, eventually it would be best if this
>> work was also done in background, somehow automatically queued by the
>> system.
>>
>> I would also recommend coming up with a usage scenario which shows a
>> problem before coding up a solution.  I believe a test with lots of
>> users doing insert and delete should eventually show the background task
>> being bogged down -- but I am not sure if moving work to additional
>> threads is much better than just spreading the work out across the
>> existing user threads.
>>
>> The code for the current background thread can be found in:
>> opensource/java/engine/org/apache/derby/impl/services/daemon
>>
>> An example of one of the unit of work put on the queue is in:
>> opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java
>>
>>
>> Dan is probably the person who most recently worked on this code, and
>> should have some comments in this area.  He should be back active on the
>> list early next week.
>>
>> Note another interesting area of research/coding would be to see how
>> derby scales on larger number of processor machines.  Not much work has
>> been done at all on machines with more than 2 processors.  The system
>> has been designed from bottom up to be multi-threaded, but not much
>> testing/monitoring has been done on 4 or more processor machines.   The
>> following single threading points exist in derby:
>>    o each user query is executed by a single thread.
>>    o the locking system in protected by a single java synchonization
>> point.
>>    o copying log records into the log is a single sync point
>>    o finding a buffer in the buffer cache is a single sync point
>>
>> All of these seemed to be reasonable designs for 1, 2 and 4 way
>> machines.
>>
>> /mikem
>>
>>
>> David Van Couvering wrote:
>>
>>  
>>
>>> I noticed on the todo list there is a need to have more than one
>>> background thread to enable better scalability with lots of client
>>> connections.  I'm trying to find a way to gently work my way into doing
>>> some work on Derby, and this seemed like a project of small enough
>>> scope
>>> to get my feet wet.  Is there any background on this, or should I just
>>> jump right in?  I didn't see any discussion of this on the list...
>>>
>>> Thanks,
>>>
>>> David
>>>
>>>   
>>
>
>
>

Re: more background threads

[David Van Couvering <Da...@Sun.COM>]

Hi, Mike, thanks for the response and very helpful overview.  At first 
blush it seems like the single daemon could easily be converted to a 
thread pool approach where work is posted to a "dispatcher" who grabs a 
thread and dispatches the work to it.  I say this without having yet 
looked at the code, but in the meantime any reasons why this obviously 
won't work would be much appreciated.

I can work on building up a test case that fills up the background 
thread so we can "prove" that whatever solution we come up with helps 
the system scale better.  I can post a test plan prior to actually 
creating the test to see if you all agree the test looks to be what we 
want it to be.
 
I can also look into testing Derby scalability on a 4-way or 8-way 
machine, I think some of these are available in our lab.  I would also 
like to do some testing on some of Sun's new multi-core chips, where you 
have 8 threads per core and 4-8 cores per CPU.  Derby seems to be 
well-suited to this architecture but it would be good to see if there 
are any gotchas.  Again, I would proposed these as plans first and get 
your feedback.

What protocol do I use to sort of "identify" this is a sub-project and 
track its progress?  Do I create a JIRA item labelled as an 
"improvement" and assign it to myself?

Thanks,

David

Mike Matrigali wrote:

>I have changed the subject, as I completely missed the original post
>which had something to do with adding Junit tests.
>
>I am not sure what is the right solution here, but getting a discussion
>going would be good.
>
>Currently a number of store actions are queued in "post commit" mode,
>which means they should be executed until after the transaction which
>queued them commits.  Currently there is one background thread which
>processes these, if it gets too full then the work is done by the actual
>thread which queued the work.   Most of the post commit work involves
>claiming space from deleted rows after their transaction commits.
>
>Going forward there is going to be a need for more background work.  I
>soon will be posting the first phase of work to allow for returning
>space back to the operating system, eventually it would be best if this
>work was also done in background, somehow automatically queued by the
>system.
>
>I would also recommend coming up with a usage scenario which shows a
>problem before coding up a solution.  I believe a test with lots of
>users doing insert and delete should eventually show the background task
>being bogged down -- but I am not sure if moving work to additional
>threads is much better than just spreading the work out across the
>existing user threads.
>
>The code for the current background thread can be found in:
>opensource/java/engine/org/apache/derby/impl/services/daemon
>
>An example of one of the unit of work put on the queue is in:
>opensource/java/engine/org/apache/derby/impl/store/access/heap/heappostcommit.java
>
>Dan is probably the person who most recently worked on this code, and
>should have some comments in this area.  He should be back active on the
>list early next week.
>
>Note another interesting area of research/coding would be to see how
>derby scales on larger number of processor machines.  Not much work has
>been done at all on machines with more than 2 processors.  The system
>has been designed from bottom up to be multi-threaded, but not much
>testing/monitoring has been done on 4 or more processor machines.   The
>following single threading points exist in derby:
>    o each user query is executed by a single thread.
>    o the locking system in protected by a single java synchonization point.
>    o copying log records into the log is a single sync point
>    o finding a buffer in the buffer cache is a single sync point
>
>All of these seemed to be reasonable designs for 1, 2 and 4 way machines.
>
>/mikem
>
>
>David Van Couvering wrote:
>
>  
>
>>I noticed on the todo list there is a need to have more than one
>>background thread to enable better scalability with lots of client
>>connections.  I'm trying to find a way to gently work my way into doing
>>some work on Derby, and this seemed like a project of small enough scope
>>to get my feet wet.  Is there any background on this, or should I just
>>jump right in?  I didn't see any discussion of this on the list...
>>
>>Thanks,
>>
>>David
>>
>>    
>>