You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Pavel Bortnovskiy <pb...@jefferies.com> on 2012/12/06 22:03:57 UTC
concurrent execution of Statement's .addBatch() and .executeBatch()
Hello:
Is it safe to call .addBatch() and .executeBatch() methods from multiple threads on the same PreparedStatement?
Simple example: batching and executing a large number of the same statements (with the same PreparedStatement) by using ExecutorService.
Thank you,
Pavel.
Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
RE: concurrent execution of Statement's .addBatch() and .executeBatch()
Posted by Raymond Kroeker <ra...@gmail.com>.
Your approach is good! Definitely caching is a good idea. You might look
at your connection pool to see if it can also pool your statements to
achieve the gains you want but also the safety you need.
Raymond
On Dec 7, 2012 7:55 AM, "Pavel Bortnovskiy" <pb...@jefferies.com>
wrote:
> Thank you, Knut.
>
> Yes, my app is doing precisely that, so I may have to use a single thread
> to operate on the same PreparedStatement, or "serialize" access to it that
> only one thread may set objects and execute.
>
> May I ask a related question, then?
>
> It has been my assumption that creation of PreparedStatement is expensive.
> Thus I create cached Prepared Statements (such as INSERT, UPDATE, DELETE,
> SELECT COUNT(*) FROM <table>) and then use them. Would it be an improvement
> in my design to create multiple PreparedStatements for those (assuring
> proper addition of objects and execution)? Would you still recommend
> caching and reusing them or creating them anew?
>
> For instance, if I have 3,000 records to insert, I could split them into
> 6x500 batches, create 6 PreparedStatements and have each of them execute
> their corresponding batches in parallel.
>
> Thank you,
> Pavel.
>
> -----Original Message-----
> From: Knut Anders Hatlen [mailto:knut.hatlen@oracle.com]
> Sent: Friday, December 07, 2012 7:01 AM
> To: Derby Discussion
> Subject: Re: concurrent execution of Statement's .addBatch() and
> .executeBatch()
>
> Pavel Bortnovskiy <pb...@jefferies.com> writes:
>
> > Hello:
> >
> > Is it safe to call .addBatch() and .executeBatch() methods from
> > multiple threads on the same PreparedStatement?
> >
> > Simple example: batching and executing a large number of the same
> > statements (with the same PreparedStatement) by using ExecutorService.
>
> Hi Pavel,
>
> Both addBatch() and executeBatch() do their work in a block synchronized
> on the connection, so in theory it should work to have many threads adding
> batches to the same PreparedStatement. I don't think it has been heavily
> tested, though.
>
> And if the threads set any parameters on the PreparedStatement, your
> application needs to synchronize the threads manually. For example, if two
> threads call
>
> ps.setInt(1, id);
> ps.setObject(2, value);
> ps.addBatch();
>
> on the same statement, you need to add synchronization to ensure that the
> addBatch() call in one thread doesn't end up using one or more parameter
> values set by the other thread.
>
> --
> Knut Anders
>
> Jefferies archives and monitors outgoing and incoming e-mail. The contents
> of this email, including any attachments, are confidential to the ordinary
> user of the email address to which it was addressed. If you are not the
> addressee of this email you may not copy, forward, disclose or otherwise
> use it or any part of it in any form whatsoever. This email may be produced
> at the request of regulators or in connection with civil litigation.
> Jefferies accepts no liability for any errors or omissions arising as a
> result of transmission. Use by other than intended recipients is
> prohibited. In the United Kingdom, Jefferies operates as Jefferies
> International Limited; registered in England: no. 1978621; registered
> office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies
> International Limited is authorised and regulated by the Financial Services
> Authority.
>
Re: concurrent execution of Statement's .addBatch() and .executeBatch()
Posted by Dag Wanvik <da...@oracle.com>.
On 10.12.2012 20:56, Pavel Bortnovskiy wrote:
> Does Derby use some kind of PrepStatement pooling?
In embedded mode, i.e. the database engine, the execution plan for
prepared statements is cached, but the prepared statement objects as
such are not pooled.
http://db.apache.org/derby/docs/10.9/tuning/ctundepth32379.html
The client driver has client side caching of prepared statements,
though, cf.
http://db.apache.org/derby/docs/10.9/adminguide/cadminnsdatasources.html.
Thanks,
Dag
RE: concurrent execution of Statement's .addBatch() and
.executeBatch()
Posted by Pavel Bortnovskiy <pb...@jefferies.com>.
Does Derby use some kind of PrepStatement pooling?
-----Original Message-----
From: Dag Wanvik [mailto:dag.wanvik@oracle.com]
Sent: Monday, December 10, 2012 2:53 PM
To: derby-user@db.apache.org
Subject: Re: concurrent execution of Statement's .addBatch() and .executeBatch()
On 07.12.2012 16:53, Pavel Bortnovskiy wrote:
> Thank you, Knut.
> It has been my assumption that creation of PreparedStatement is expensive. Thus I create cached Prepared Statements (such as INSERT, UPDATE, DELETE, SELECT COUNT(*) FROM <table>) and then use them. Would it be an improvement in my design to create multiple PreparedStatements for those (assuring proper addition of objects and execution)? Would you still recommend caching and reusing them or creating them anew?
Please also note that as long as the statements are textually the same, i.e. arguments are given dynamically as "?" , Derby is able to use its internal cache to avoid SQL compilation for more than the first thread, even if each of the thread has a created a separate PreparedStatement in its own connection. Cf.
http://db.apache.org/derby/docs/10.9/ref/rrefproperstatementcachesize.html
Thanks,
Dag
>
> For instance, if I have 3,000 records to insert, I could split them into 6x500 batches, create 6 PreparedStatements and have each of them execute their corresponding batches in parallel.
>
> Thank you,
> Pavel.
>
> -----Original Message-----
> From: Knut Anders Hatlen [mailto:knut.hatlen@oracle.com]
> Sent: Friday, December 07, 2012 7:01 AM
> To: Derby Discussion
> Subject: Re: concurrent execution of Statement's .addBatch() and
> .executeBatch()
>
> Pavel Bortnovskiy <pb...@jefferies.com> writes:
>
>> Hello:
>>
>> Is it safe to call .addBatch() and .executeBatch() methods from
>> multiple threads on the same PreparedStatement?
>>
>> Simple example: batching and executing a large number of the same
>> statements (with the same PreparedStatement) by using ExecutorService.
> Hi Pavel,
>
> Both addBatch() and executeBatch() do their work in a block synchronized on the connection, so in theory it should work to have many threads adding batches to the same PreparedStatement. I don't think it has been heavily tested, though.
>
> And if the threads set any parameters on the PreparedStatement, your
> application needs to synchronize the threads manually. For example, if
> two threads call
>
> ps.setInt(1, id);
> ps.setObject(2, value);
> ps.addBatch();
>
> on the same statement, you need to add synchronization to ensure that the addBatch() call in one thread doesn't end up using one or more parameter values set by the other thread.
>
> --
> Knut Anders
>
> Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
Re: concurrent execution of Statement's .addBatch() and .executeBatch()
Posted by Dag Wanvik <da...@oracle.com>.
On 07.12.2012 16:53, Pavel Bortnovskiy wrote:
> Thank you, Knut.
> It has been my assumption that creation of PreparedStatement is expensive. Thus I create cached Prepared Statements (such as INSERT, UPDATE, DELETE, SELECT COUNT(*) FROM <table>) and then use them. Would it be an improvement in my design to create multiple PreparedStatements for those (assuring proper addition of objects and execution)? Would you still recommend caching and reusing them or creating them anew?
Please also note that as long as the statements are textually the same,
i.e. arguments are given dynamically as "?" , Derby is able to use its
internal cache to avoid SQL compilation for more than the first thread,
even if each of the thread has a created a separate PreparedStatement in
its own connection. Cf.
http://db.apache.org/derby/docs/10.9/ref/rrefproperstatementcachesize.html
Thanks,
Dag
>
> For instance, if I have 3,000 records to insert, I could split them into 6x500 batches, create 6 PreparedStatements and have each of them execute their corresponding batches in parallel.
>
> Thank you,
> Pavel.
>
> -----Original Message-----
> From: Knut Anders Hatlen [mailto:knut.hatlen@oracle.com]
> Sent: Friday, December 07, 2012 7:01 AM
> To: Derby Discussion
> Subject: Re: concurrent execution of Statement's .addBatch() and .executeBatch()
>
> Pavel Bortnovskiy <pb...@jefferies.com> writes:
>
>> Hello:
>>
>> Is it safe to call .addBatch() and .executeBatch() methods from
>> multiple threads on the same PreparedStatement?
>>
>> Simple example: batching and executing a large number of the same
>> statements (with the same PreparedStatement) by using ExecutorService.
> Hi Pavel,
>
> Both addBatch() and executeBatch() do their work in a block synchronized on the connection, so in theory it should work to have many threads adding batches to the same PreparedStatement. I don't think it has been heavily tested, though.
>
> And if the threads set any parameters on the PreparedStatement, your application needs to synchronize the threads manually. For example, if two threads call
>
> ps.setInt(1, id);
> ps.setObject(2, value);
> ps.addBatch();
>
> on the same statement, you need to add synchronization to ensure that the addBatch() call in one thread doesn't end up using one or more parameter values set by the other thread.
>
> --
> Knut Anders
>
> Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
RE: concurrent execution of Statement's .addBatch() and
.executeBatch()
Posted by Pavel Bortnovskiy <pb...@jefferies.com>.
Thank you, Knut.
Yes, my app is doing precisely that, so I may have to use a single thread to operate on the same PreparedStatement, or "serialize" access to it that only one thread may set objects and execute.
May I ask a related question, then?
It has been my assumption that creation of PreparedStatement is expensive. Thus I create cached Prepared Statements (such as INSERT, UPDATE, DELETE, SELECT COUNT(*) FROM <table>) and then use them. Would it be an improvement in my design to create multiple PreparedStatements for those (assuring proper addition of objects and execution)? Would you still recommend caching and reusing them or creating them anew?
For instance, if I have 3,000 records to insert, I could split them into 6x500 batches, create 6 PreparedStatements and have each of them execute their corresponding batches in parallel.
Thank you,
Pavel.
-----Original Message-----
From: Knut Anders Hatlen [mailto:knut.hatlen@oracle.com]
Sent: Friday, December 07, 2012 7:01 AM
To: Derby Discussion
Subject: Re: concurrent execution of Statement's .addBatch() and .executeBatch()
Pavel Bortnovskiy <pb...@jefferies.com> writes:
> Hello:
>
> Is it safe to call .addBatch() and .executeBatch() methods from
> multiple threads on the same PreparedStatement?
>
> Simple example: batching and executing a large number of the same
> statements (with the same PreparedStatement) by using ExecutorService.
Hi Pavel,
Both addBatch() and executeBatch() do their work in a block synchronized on the connection, so in theory it should work to have many threads adding batches to the same PreparedStatement. I don't think it has been heavily tested, though.
And if the threads set any parameters on the PreparedStatement, your application needs to synchronize the threads manually. For example, if two threads call
ps.setInt(1, id);
ps.setObject(2, value);
ps.addBatch();
on the same statement, you need to add synchronization to ensure that the addBatch() call in one thread doesn't end up using one or more parameter values set by the other thread.
--
Knut Anders
Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorised and regulated by the Financial Services Authority.
Re: concurrent execution of Statement's .addBatch() and .executeBatch()
Posted by Knut Anders Hatlen <kn...@oracle.com>.
Pavel Bortnovskiy <pb...@jefferies.com> writes:
> Hello:
>
> Is it safe to call .addBatch() and .executeBatch() methods from
> multiple threads on the same PreparedStatement?
>
> Simple example: batching and executing a large number of the same
> statements (with the same PreparedStatement) by using ExecutorService.
Hi Pavel,
Both addBatch() and executeBatch() do their work in a block synchronized
on the connection, so in theory it should work to have many threads
adding batches to the same PreparedStatement. I don't think it has been
heavily tested, though.
And if the threads set any parameters on the PreparedStatement, your
application needs to synchronize the threads manually. For example, if
two threads call
ps.setInt(1, id);
ps.setObject(2, value);
ps.addBatch();
on the same statement, you need to add synchronization to ensure that
the addBatch() call in one thread doesn't end up using one or more
parameter values set by the other thread.
--
Knut Anders