You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Michael Pflueger <Mi...@sma.de> on 2011/06/07 13:38:08 UTC
concurrent thread exception plus deadlock on delete statement using
slice
I've got some trouble executing delete statements using OpenJPA 2.1.0 with slice and 2 MySQL 5 backends.
Here's my code:
em.getTransaction().begin();
em.createQuery("delete from A").executeUpdate();
em.createQuery("delete from B").executeUpdate();
em.createQuery("delete from C").executeUpdate();
em.getTransaction().commit();
With this, I get the "Multiple concurrent threads attempted to access a single broker" Exception,
Even though my application only has one thread at this point.
Out of curiosity I enabled "Openjpa.Multithreaded", yet with this, even a single delete statement leads to a deadlock.
Has anybody got this working or can provide some help?
regards,
Michael
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
The error shows that the delete failed on the database. The visible
threading error message looks to be on the exception path.
I am investigating but so far I am not been able to reproduce the error. A
reproducible JUnit test will help.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6456167.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Michael Pflueger <Mi...@sma.de>.
Hi,
thanks for your help.
I did some further tests today. The problem already appears when I want to delete entries of an already emtpy table of a simple test entity with only a primary key as property.
I don't really believe in database errors yet, because I haven't seen any in my MySQL logs, and when I manually enter the query OpenJPA sends to MySQL, it works.
Also the fact that I don't get any exception but a deadlock when I enable openjpa.Multithreaded somehow hints to a threading problem I think.
With openjpa.Multithreaded enabled, my application thread and both slice threads seem to wait for a lock, I can post their stacktraces below.
Now I fiddled a bit with my configuration and noticed something, I had openjpa.BrokerImpl set to non-finalizing.
When I comment out this property, the exception (or the deadlock with multithreaded mode) is gone, and everything seems to work, at first glance.
Then however I noticed it does not delete the table, and checked my MySQL logs.
OpenJPA sends the delete table commands, but does not commit the change, instead it issues a rollback only.
I have no idea why, and the OpenJPA log (with loglevel set to TRACE) shows no exception, nothing that hints at a rollback.
I'll paste some log excerpts logs below.
I'm really puzzled.
Do you have any further ideas what I could try?
If not, I'll indeed try to create a minimal testcase to reproduce the problem tomorrow.
Just a word about my System btw, I'm on a dualcore CPU running Windows XP and Java 1.6.0_25.
Regards,
Michael
Stacktraces of the deadlock when in multithreaded mode:
My application thread:
Thread [main] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: not available
FutureTask$Sync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: not available
FutureTask$Sync(AbstractQueuedSynchronizer).doAcquireSharedInterruptibly(int) line: not available
FutureTask$Sync(AbstractQueuedSynchronizer).acquireSharedInterruptibly(int) line: not available
FutureTask$Sync.innerGet() line: not available
FutureTask<V>.get() line: not available
DistributedStoreQuery$ParallelExecutor.executeDelete(StoreQuery, Object[]) line: 219
QueryImpl.delete(StoreQuery, StoreQuery$Executor, Object[]) line: 1025
QueryImpl.execute(int, Object[]) line: 822
QueryImpl.deleteAll(Object[]) line: 888
QueryImpl.deleteAll() line: 884
DelegatingQuery.deleteAll() line: 558
QueryImpl<X>.executeUpdate() line: 362
CorrectionDemo.main(String[]) line: 25
Main-slice-1:
Thread [main-slice-1] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: not available
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: not available
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquireQueued(AbstractQueuedSynchronizer$Node, int) line: not available
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquire(int) line: not available
ReentrantLock$NonfairSync.lock() line: not available
ReentrantLock.lock() line: not available
BrokerImpl.lock() line: 4398
BrokerImpl.beginOperation(boolean) line: 1907
BrokerImpl.beginStore() line: 1390
JDBCStoreQuery.executeBulkOperation(ClassMetaData[], boolean, ExpressionFactory[], QueryExpressions[], Object[], Map) line: 566
JDBCStoreQuery.executeDelete(StoreQuery$Executor, ClassMetaData, ClassMetaData[], boolean, ExpressionFactory[], QueryExpressions[], Object[]) line: 491
ExpressionStoreQuery$DataStoreExecutor.executeDelete(StoreQuery, Object[]) line: 788
DistributedStoreQuery$DeleteExecutor.call() line: 283
DistributedStoreQuery$DeleteExecutor.call() line: 277
FutureTask$Sync.innerRun() line: not available
FutureTask<V>.run() line: not available
ThreadPoolExecutor$Worker.runTask(Runnable) line: not available
ThreadPoolExecutor$Worker.run() line: not available
SliceThread(Thread).run() line: not available
Main-slice-0:
Thread [main-slice-0] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: not available
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: not available
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquireQueued(AbstractQueuedSynchronizer$Node, int) line: not available
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquire(int) line: not available
ReentrantLock$NonfairSync.lock() line: not available
ReentrantLock.lock() line: not available
BrokerImpl.lock() line: 4398
BrokerImpl.beginOperation(boolean) line: 1907
BrokerImpl.beginStore() line: 1390
JDBCStoreQuery.executeBulkOperation(ClassMetaData[], boolean, ExpressionFactory[], QueryExpressions[], Object[], Map) line: 566
JDBCStoreQuery.executeDelete(StoreQuery$Executor, ClassMetaData, ClassMetaData[], boolean, ExpressionFactory[], QueryExpressions[], Object[]) line: 491
ExpressionStoreQuery$DataStoreExecutor.executeDelete(StoreQuery, Object[]) line: 788
DistributedStoreQuery$DeleteExecutor.call() line: 283
DistributedStoreQuery$DeleteExecutor.call() line: 277
FutureTask$Sync.innerRun() line: not available
FutureTask<V>.run() line: not available
ThreadPoolExecutor$Worker.runTask(Runnable) line: not available
ThreadPoolExecutor$Worker.run() line: not available
SliceThread(Thread).run() line: not available
OpenJPA log with "non-finalizing" commented out:
4125 openjpa TRACE [main] openjpa.Query - Executing query: delete from PlantCorrection pc
4109 openjpa.One TRACE [main-slice-1] openjpa.jdbc.SQL - <t 22245100, conn 964388> executing prepstmnt 31098843 DELETE t0 FROM PlantCorrection t0
4109 openjpa.One TRACE [main-slice-1] openjpa.jdbc.SQL - <t 22245100, conn 964388> [0 ms] spent
4109 openjpa.One TRACE [main-slice-1] openjpa.jdbc.JDBC - <t 22245100, conn 0> [0 ms] close
3859 openjpa.Two TRACE [main-slice-0] openjpa.jdbc.SQL - <t 13969258, conn 9924771> executing prepstmnt 19874213 DELETE t0 FROM PlantCorrection t0
3859 openjpa.Two TRACE [main-slice-0] openjpa.jdbc.SQL - <t 13969258, conn 9924771> [0 ms] spent
3859 openjpa.Two TRACE [main-slice-0] openjpa.jdbc.JDBC - <t 13969258, conn 0> [0 ms] close
MySQL log:
1689 Query SET autocommit=0
1689 Query DELETE t0 FROM PlantCorrection t0
1689 Query rollback
1689 Query SET autocommit=1
1691 Query SET autocommit=0
1691 Query DELETE t0 FROM PlantCorrection t0
1691 Query rollback
1691 Query SET autocommit=1
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Pinaki Poddar <pp...@apache.org>.
Mike,
This one would require time to fix. I do not have bandwidth to touch this
one right now, nor do I have access to solid testing infrastructure to
validate my changes against risk of regression.
Please file a JIRA and hopefully someone (including you:) may supply a
patch.
For a quick fix, one may try this route in
...openjpa.persistence.PersistenceExceptions$RuntimeExceptionTranslator {
- public RuntimeException translate(RuntimeException re)
+ public synchronized RuntimeException translate(RuntimeException re)
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6497485.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Michael Pflueger <Mi...@sma.de>.
Hi,
any news on this one, or are there things I could test further to get around this rollback/no commit problem?
-----Ursprüngliche Nachricht-----
Von: Michael Pflueger [mailto:Michael.Pflueger@sma.de]
Gesendet: Dienstag, 14. Juni 2011 09:15
An: users@openjpa.apache.org
Betreff: Re: concurrent thread exception plus deadlock on delete statement using slice
Hi,
that solves the deadlock and exception problems, but when I remove that line from my configuration, I still have that rollback issue,
that is OpenJPA sends the delete statements to the database but issues a rollback only, no commit (see my example project with the "ROLLBACK_persistence.xml" configuration where the BrokerImpl property is commented out)
Do you have any idea why this is so?
Regards,
Mike
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Michael Pflueger <Mi...@sma.de>.
Hi,
that solves the deadlock and exception problems, but when I remove that line from my configuration, I still have that rollback issue,
that is OpenJPA sends the delete statements to the database but issues a rollback only, no commit (see my example project with the "ROLLBACK_persistence.xml" configuration where the BrokerImpl property is commented out)
Do you have any idea why this is so?
Regards,
Mike
-----Ursprüngliche Nachricht-----
Von: Pinaki Poddar [mailto:ppoddar@apache.org]
Gesendet: Montag, 13. Juni 2011 22:35
An: users@openjpa.apache.org
Betreff: Re: concurrent thread exception plus deadlock on delete statement using slice
Hi,
Slice uses its own Broker (DistributedBrokerImpl) implementaion. Whereas
your configuration tries to overwrite that with
<property name="openjpa.BrokerImpl" value="non-finalizing"></property>
For a quick turnaround, I would advise that you remove that from your
configuration.
If the problem persists still, I will examine it further.
I will add this caveat/limitation in our documentation.
Regards --
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6471718.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
Slice uses its own Broker (DistributedBrokerImpl) implementaion. Whereas
your configuration tries to overwrite that with
<property name="openjpa.BrokerImpl" value="non-finalizing"></property>
For a quick turnaround, I would advise that you remove that from your
configuration.
If the problem persists still, I will examine it further.
I will add this caveat/limitation in our documentation.
Regards --
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6471718.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Michael Pflueger <Mi...@sma.de>.
Hi,
attached is a little zipped eclipse project excluding libs to demonstrate the problem.
Libs I'm using are openjpa-all-2.1.0.jar and mysql-connector-java-5.1.16-bin.jar
I have included 3 persistence.xml files, one showing the exception, one the deadlock and one the rollback scenario.
I hope this helps and you can reproduce this.
regards,
Michael
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
Here is a possible diagnosis:
First, a little background:
When exceptions are thrown at the database operation, that internal
exception bubbles up to the facade (i.e. EntityManager). The facade
translates the exception to an application visible exception such as
OptimisticException or EntityNotFoundException. Depending on the severity of
the exception, the transaction is set rollback only mode.
In this case,
1. Multiple slices/shards are throwing database exceptions.
2. In exception translation process gets very confused because it is
unaware that multiple slices are throwing exceptions from different threads.
I do not have any immediate remedy. Moreover, I can not think of creating
a delete query that reliably fails at database (a query with a wrong syntax
or non-existent table would not do, because the query parser would throw the
error not the database, and parsing happens only once in Slice).
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6456295.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Michael Pflueger <Mi...@sma.de>.
To give you some more info, when I have an almost empty table with only one entity, it actually works most of the time, but still crashes sometimes.
Most of our entities are replicated, I'm not sure if this matters and whether it also happens with distributed entities, I could check that if you like.
Regards,
Michael
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Michael Pflueger <Mi...@sma.de>.
Hello,
> 1. Try em.createQuery("delete from A a")
I've tried this, but the error persists. The trace is:
Exception in thread "main" <openjpa-2.1.0-r422266:1071316 fatal general error> org.apache.openjpa.persistence.PersistenceException: Multiple concurrent threads attempted to access a single broker. By default brokers are not thread safe; if you require and/or intend a broker to be accessed by more than one thread, set the openjpa.Multithreaded property to true to override the default behavior.
at org.apache.openjpa.kernel.BrokerImpl.endOperation(BrokerImpl.java:1935)
at org.apache.openjpa.kernel.BrokerImpl.isActive(BrokerImpl.java:1883)
at org.apache.openjpa.kernel.DelegatingBroker.isActive(DelegatingBroker.java:468)
at org.apache.openjpa.persistence.EntityManagerImpl.isActive(EntityManagerImpl.java:668)
at org.apache.openjpa.persistence.PersistenceExceptions$2.translate(PersistenceExceptions.java:76)
at org.apache.openjpa.kernel.DelegatingQuery.translate(DelegatingQuery.java:99)
at org.apache.openjpa.kernel.DelegatingQuery.deleteAll(DelegatingQuery.java:560)
at org.apache.openjpa.persistence.QueryImpl.executeUpdate(QueryImpl.java:362)
at betriebsdaten.demo.OpenJpaDemo.main(OpenJpaDemo.java:28)
___________________________________________________
SMA Solar Technology AG
Aufsichtsrat: Guenther Cramer (Vorsitzender)
Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
Handelsregister: Amtsgericht Kassel HRB 3972
Sitz der Gesellschaft: 34266 Niestetal
USt-ID-Nr. DE 113 08 59 54
WEEE-Reg.-Nr. DE 95881150
___________________________________________________
Re: concurrent thread exception plus deadlock on delete statement
using slice
Posted by Pinaki Poddar <pp...@apache.org>.
Hi,
> em.createQuery("delete from A")
1. Try em.createQuery("delete from A a")
2. Please post the stacktrace, if the error persists
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6450253.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.