You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Claudio Di Vita <c....@gmail.com> on 2009/08/03 14:58:36 UTC

Persist issue in multithreaded environment

Hi to all,

I'm having some troubles using OpenJPA in a multithreaded environment.

Sometimes (because it happens randomly) some entities aren't persisted, but
OpenJPA doesn't throw any exception. Please note that I persist a set of
entities (instances of the same class) that don't share any kind of
information.

I haven't set the property openjpa.Multithreaded to true because I use a
different EntityManager for each thread (I get it through a ThreadLocal).
Obviously If I set this property everything seems to work right.

So I want to understand why I have to set this property: I'm not doing
something right or property openjpa.Multithreaded is mandatory for
multithreaded environments (even using different EntityManagers) ??

Thank you for the help,

Claudio

-----
Not everything that can be counted counts, and not everything that counts can
be counted - Albert Einstein
-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3377510.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Persist issue in multithreaded environment

Posted by Jean-Baptiste BRIAUD -- Novlog <j-...@novlog.com>.
On Aug 3, 2009, at 15:43 , Claudio Di Vita wrote:

> Jean-Baptiste BRIAUD -- Novlog wrote:
>>
>> Did you compared using == what you think is different instance of
>> EntityManager ?
>> Reading your message tend to prove that EntityManager is shared and
>> when you think you got a new one it is not a new one.
>>
>
> Why I have to check an instance returned by a threadlocal variable ??
>
Because TheadLocal used in certain way guarantee that a variable will  
be local to a thread, as its name impled, but doesn't guarantee that  
this instance will be different for each thread.
The answer to your question is not in ThreadLocal class but inside the  
createEntityManager() method.


> I use the following Th ThreadLocal<EntityManager>:
>
> private static class ThreadLocalEntityManager extends
> ThreadLocal<EntityManager> {
>
>        /* (non-Javadoc)
>         * @see java.lang.ThreadLocal#get()
>         */
>        @Override
>        public EntityManager get() {
>
>            /* Get the current entity manager */
>            EntityManager em = super.get();
>
>            /* The entity manager was closed */
>            if (!em.isOpen()) {
>
>                /* Create a new entity manager */
>                em = factory.createEntityManager();
>
>                /* Update the entity manager */
>                set(em);
>            }
>
>            return em;
>        }
>
>        /* (non-Javadoc)
>         * @see java.lang.ThreadLocal#initialValue()
>         */
>        @Override
>        protected EntityManager initialValue() {
>
>            return factory.createEntityManager();
>        }
>    }
>
> Where factory is a static EntityManagerFactory.
>
> What is going wrong ??
>
> -----
> Not everything that can be counted counts, and not everything that  
> counts can
> be counted - Albert Einstein
> -- 
> View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3377741.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>


Re: Persist issue in multithreaded environment

Posted by Jean-Baptiste BRIAUD -- Novlog <j-...@novlog.com>.
did you had a look on that method on the EntityManager :
setMultithreaded(boolean multi)
           Whether the entity manager or its managed instances are  
used in a multithreaded environment.

It sound like EntityManager has to know wheter or not it is used with  
threads.
Probably the equivalent of the properties you first talked about.

Hope this helps.

Re: Persist issue in multithreaded environment

Posted by Claudio Di Vita <c....@gmail.com>.

Kevin Sutter wrote:
> 
> 
> Claudio Di Vita wrote:
>> 
>> I turned on all TRACE messages, but I cannot see any exception. The SQL
>> is
>> generated for all records, but I cannot see them as rows in tables.
>> 
> 
>  Interesting.  We don't generate and log the SQL until it's ready to be
> sent
> to the database.  So, it's very strange that the TRACE all looks okay but
> the data is not getting to the database.  Have you tried tracing your
> database to see whether all records are getting there and whether any
> exceptions are getting swallowed on that end?
> 
> It's almost as if the SQL is getting generated and maybe even flushed to
> the
> database, but then the transaction is getting rolled back for some reason.
> Not making a lot of sense...
> 
> Is it possible to share your TRACE file with a bit of explanation of how
> to
> interpret your processing?
> 
Maybe the problem occurs when the SQL is sent to the DB. For example,
concurrent invocations may override the statements.

I can give you the TRACE file, but before I have to remove sensible data
(and I need some time, the file size if around 300 MB)

-----
Not everything that counts can be counted - Albert Einstein

http://www.linkedin.com/in/cdivita
-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3397982.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Persist issue in multithreaded environment

Posted by Kevin Sutter <kw...@gmail.com>.
Hi Claudio,
More comments inline below...

On Wed, Aug 5, 2009 at 3:14 AM, Claudio Di Vita <c....@gmail.com> wrote:

>
> Kevin Sutter wrote:
> >
> > Hi Claudio,
> > You mention in an earlier post that when you turn on SQL Trace that you
> > see
> > all of the Inserts getting created.  So, does that mean that if you turn
> > Trace on that it seems to affect the processing and everything works as
> > expected?  Or, do you see SQL in the Trace that is supposedly generated
> > and
> > executed, but it doesn't result in rows in tables?
> >
> > Have you turned on all of the TRACE to see if there are any errant
> > exceptions not being surfaced?  (If turning Trace on affects the ability
> > to
> > reproduce the problem, then this may not be a valid request.)
> >
> I turned on all TRACE messages, but I cannot see any exception. The SQL is
> generated for all records, but I cannot see them as rows in tables.
>

 Interesting.  We don't generate and log the SQL until it's ready to be sent
to the database.  So, it's very strange that the TRACE all looks okay but
the data is not getting to the database.  Have you tried tracing your
database to see whether all records are getting there and whether any
exceptions are getting swallowed on that end?

It's almost as if the SQL is getting generated and maybe even flushed to the
database, but then the transaction is getting rolled back for some reason.
Not making a lot of sense...

Is it possible to share your TRACE file with a bit of explanation of how to
interpret your processing?

Kevin Sutter wrote:
> >
> > In your thread local method, you check if the EM is Open.  If it's
> Closed,
> > then you create a new one.  If it's Open, then you just continue to use
> > it.
> > Can you further explain your lifecycle processing for the EM?  That is,
> > who
> > is responsible for Closing the EM when it's done?  And, under what
> > conditions?  What transaction demarcation are you using?  Local
> > EntityTransactions or some external transaction manager?  I'm just
> > wondering
> > if there's some hiccup between the opening/closing of the EM and the
> > transaction demarcation processing.
> >
> > BTW, I agree with Craig's earlier comment that there should be no reason
> > to
> > use the Multihhreaded property if you are only using a single EM per
> > thread.  And, if you set this openjpa.Multithreaded to "true", then the
> > problem does not surface?  If that's the case, then it seems that you are
> > accidentally using an EM by more than one thread.  I just don't see how
> > yet...
> >
>
> The EM provided by the ThreadLocal is used by some utility methods that
> perform persistence operations and then close the EM. So I have to check
> the
> state of the EM stored in the ThreadLocal, but there aren't "pending" EM.
>
> I use local transactions...maybe I should try with a standalone JTA
> provider
> ??


Well, this might be easier said than done.  By an external JTA provider, I
meant one provided with an application server.  I am not aware of a JTA
provider that runs in JSE.

The local transactions should be just fine.  I'm just wondering whether
these local transactions are getting properly committed before closing of
the corresponding EntityManager.  This might explain some of the unexpected
rollbacks that I was guessing at above...

Kevin


>
>
>
> -----
> Not everything that can be counted counts - Albert Einstein
>
> http://www.linkedin.com/in/cdivita
> --
> View this message in context:
> http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3389868.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>

Re: Persist issue in multithreaded environment

Posted by Claudio Di Vita <c....@gmail.com>.

Kevin Sutter wrote:
> 
> Hi Claudio,
> You mention in an earlier post that when you turn on SQL Trace that you
> see
> all of the Inserts getting created.  So, does that mean that if you turn
> Trace on that it seems to affect the processing and everything works as
> expected?  Or, do you see SQL in the Trace that is supposedly generated
> and
> executed, but it doesn't result in rows in tables?
> 
> Have you turned on all of the TRACE to see if there are any errant
> exceptions not being surfaced?  (If turning Trace on affects the ability
> to
> reproduce the problem, then this may not be a valid request.)
> 
I turned on all TRACE messages, but I cannot see any exception. The SQL is
generated for all records, but I cannot see them as rows in tables.


Kevin Sutter wrote:
> 
> In your thread local method, you check if the EM is Open.  If it's Closed,
> then you create a new one.  If it's Open, then you just continue to use
> it.
> Can you further explain your lifecycle processing for the EM?  That is,
> who
> is responsible for Closing the EM when it's done?  And, under what
> conditions?  What transaction demarcation are you using?  Local
> EntityTransactions or some external transaction manager?  I'm just
> wondering
> if there's some hiccup between the opening/closing of the EM and the
> transaction demarcation processing.
> 
> BTW, I agree with Craig's earlier comment that there should be no reason
> to
> use the Multihhreaded property if you are only using a single EM per
> thread.  And, if you set this openjpa.Multithreaded to "true", then the
> problem does not surface?  If that's the case, then it seems that you are
> accidentally using an EM by more than one thread.  I just don't see how
> yet...
> 

The EM provided by the ThreadLocal is used by some utility methods that
perform persistence operations and then close the EM. So I have to check the
state of the EM stored in the ThreadLocal, but there aren't "pending" EM.

I use local transactions...maybe I should try with a standalone JTA provider
??


-----
Not everything that can be counted counts - Albert Einstein

http://www.linkedin.com/in/cdivita
-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3389868.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Persist issue in multithreaded environment

Posted by Kevin Sutter <kw...@gmail.com>.
Hi Claudio,
You mention in an earlier post that when you turn on SQL Trace that you see
all of the Inserts getting created.  So, does that mean that if you turn
Trace on that it seems to affect the processing and everything works as
expected?  Or, do you see SQL in the Trace that is supposedly generated and
executed, but it doesn't result in rows in tables?

Have you turned on all of the TRACE to see if there are any errant
exceptions not being surfaced?  (If turning Trace on affects the ability to
reproduce the problem, then this may not be a valid request.)

In your thread local method, you check if the EM is Open.  If it's Closed,
then you create a new one.  If it's Open, then you just continue to use it.
Can you further explain your lifecycle processing for the EM?  That is, who
is responsible for Closing the EM when it's done?  And, under what
conditions?  What transaction demarcation are you using?  Local
EntityTransactions or some external transaction manager?  I'm just wondering
if there's some hiccup between the opening/closing of the EM and the
transaction demarcation processing.

BTW, I agree with Craig's earlier comment that there should be no reason to
use the Multihhreaded property if you are only using a single EM per
thread.  And, if you set this openjpa.Multithreaded to "true", then the
problem does not surface?  If that's the case, then it seems that you are
accidentally using an EM by more than one thread.  I just don't see how
yet...

Kevin

On Tue, Aug 4, 2009 at 2:49 AM, Claudio Di Vita <c....@gmail.com> wrote:

>
>
> Craig L Russell wrote:
> >
> > The usual symptom for a multithreading problem is an exception thrown
> > inside an implementation method that can't be explained by application
> > code.
> >
> > What are your symptoms?
> >
> I persist a set of entities in n different threads at the same time. Each
> thread uses a different EntityManager.
>
> At the end of computation sometimes I have n rows in my table, sometimes I
> have n - 1, or n - 2 rows. No exception is thrown and the persist method is
> always invoked n times.
>
> Any ideas ??
>
> -----
> Not everything that can be counted counts, and not everything that counts
> can
> be counted - Albert Einstein
> --
> View this message in context:
> http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3383069.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>

Re: Persist issue in multithreaded environment

Posted by Claudio Di Vita <c....@gmail.com>.

Craig L Russell wrote:
> 
> The usual symptom for a multithreading problem is an exception thrown  
> inside an implementation method that can't be explained by application  
> code.
> 
> What are your symptoms?
> 
I persist a set of entities in n different threads at the same time. Each
thread uses a different EntityManager.

At the end of computation sometimes I have n rows in my table, sometimes I
have n - 1, or n - 2 rows. No exception is thrown and the persist method is
always invoked n times.

Any ideas ??

-----
Not everything that can be counted counts, and not everything that counts can
be counted - Albert Einstein
-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3383069.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Persist issue in multithreaded environment

Posted by Craig L Russell <Cr...@Sun.COM>.
Hi,

There should not be any need to set the Multithreaded flag to  
correctly use exactly one entity manager for a thread. The entity  
manager factory itself is thread safe.

The usual symptom for a multithreading problem is an exception thrown  
inside an implementation method that can't be explained by application  
code.

What are your symptoms?

Regards,

Craig

On Aug 3, 2009, at 6:43 AM, Claudio Di Vita wrote:

>
>
>
> Jean-Baptiste BRIAUD -- Novlog wrote:
>>
>> Did you compared using == what you think is different instance of
>> EntityManager ?
>> Reading your message tend to prove that EntityManager is shared and
>> when you think you got a new one it is not a new one.
>>
>
> Why I have to check an instance returned by a threadlocal variable ??
>
> I use the following Th ThreadLocal<EntityManager>:
>
> private static class ThreadLocalEntityManager extends
> ThreadLocal<EntityManager> {
>
>        /* (non-Javadoc)
>         * @see java.lang.ThreadLocal#get()
>         */
>        @Override
>        public EntityManager get() {
>
>            /* Get the current entity manager */
>            EntityManager em = super.get();
>
>            /* The entity manager was closed */
>            if (!em.isOpen()) {
>
>                /* Create a new entity manager */
>                em = factory.createEntityManager();
>
>                /* Update the entity manager */
>                set(em);
>            }
>
>            return em;
>        }
>
>        /* (non-Javadoc)
>         * @see java.lang.ThreadLocal#initialValue()
>         */
>        @Override
>        protected EntityManager initialValue() {
>
>            return factory.createEntityManager();
>        }
>    }
>
> Where factory is a static EntityManagerFactory.
>
> What is going wrong ??
>
> -----
> Not everything that can be counted counts, and not everything that  
> counts can
> be counted - Albert Einstein
> -- 
> View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3377741.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.

Craig L Russell
Architect, Sun Java Enterprise System http://db.apache.org/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!


Re: Persist issue in multithreaded environment

Posted by Claudio Di Vita <c....@gmail.com>.


Jean-Baptiste BRIAUD -- Novlog wrote:
> 
> Did you compared using == what you think is different instance of  
> EntityManager ?
> Reading your message tend to prove that EntityManager is shared and  
> when you think you got a new one it is not a new one.
> 

Why I have to check an instance returned by a threadlocal variable ??

I use the following Th ThreadLocal<EntityManager>:

private static class ThreadLocalEntityManager extends
ThreadLocal<EntityManager> {

        /* (non-Javadoc)
         * @see java.lang.ThreadLocal#get()
         */
        @Override
        public EntityManager get() {

            /* Get the current entity manager */
            EntityManager em = super.get();

            /* The entity manager was closed */
            if (!em.isOpen()) {

                /* Create a new entity manager */
                em = factory.createEntityManager();

                /* Update the entity manager */
                set(em);
            }

            return em;
        }

        /* (non-Javadoc)
         * @see java.lang.ThreadLocal#initialValue()
         */
        @Override
        protected EntityManager initialValue() {

            return factory.createEntityManager();
        }
    }

Where factory is a static EntityManagerFactory.

What is going wrong ??

-----
Not everything that can be counted counts, and not everything that counts can
be counted - Albert Einstein
-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3377741.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Persist issue in multithreaded environment

Posted by Jean-Baptiste BRIAUD -- Novlog <j-...@novlog.com>.
On Aug 3, 2009, at 14:58 , Claudio Di Vita wrote:

>
> Hi to all,
>
> I'm having some troubles using OpenJPA in a multithreaded environment.
>
> Sometimes (because it happens randomly) some entities aren't  
> persisted, but
> OpenJPA doesn't throw any exception. Please note that I persist a  
> set of
> entities (instances of the same class) that don't share any kind of
> information.
>
> I haven't set the property openjpa.Multithreaded to true because I  
> use a
> different EntityManager for each thread (I get it through a  
> ThreadLocal).
> Obviously If I set this property everything seems to work right.
>
> So I want to understand why I have to set this property: I'm not doing
> something right or property openjpa.Multithreaded is mandatory for
> multithreaded environments (even using different EntityManagers) ??
>
Did you compared using == what you think is different instance of  
EntityManager ?
Reading your message tend to prove that EntityManager is shared and  
when you think you got a new one it is not a new one.

> Thank you for the help,
>
> Claudio
>
> -----
> Not everything that can be counted counts, and not everything that  
> counts can
> be counted - Albert Einstein
> -- 
> View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3377510.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>


Re: Persist issue in multithreaded environment

Posted by Jean-Baptiste BRIAUD -- Novlog <j-...@novlog.com>.
On Aug 3, 2009, at 18:20 , Claudio Di Vita wrote:

>
>
> Jean-Baptiste BRIAUD -- Novlog wrote:
>>
>> The answer to your question is not in ThreadLocal class but inside  
>> the
>> createEntityManager() method.
>>
> So you think that OpenJPA implementation of EntityManagerFactory is  
> not
> thread-safe ??

I would reformulate as following : apparently, you found yourself that  
OpenKPA need a parameter to be thread safe :-)



Re: Persist issue in multithreaded environment

Posted by Claudio Di Vita <c....@gmail.com>.

Jean-Baptiste BRIAUD -- Novlog wrote:
> 
> The answer to your question is not in ThreadLocal class but inside the  
> createEntityManager() method.
> 
So you think that OpenJPA implementation of EntityManagerFactory is not
thread-safe ?? I don't catch any exception related to transactions, moreover
when I activate SQL trace I can see an INSERT statement for each record !!!


Rick Curtis wrote:
> 
> This is a bit of a stretch, but do you have a Version field [1] on each of
> your Entities?
> 
> [1]
> http://openjpa.apache.org/builds/latest/docs/manual/manual.html#jpa_overview_pc_version
> 
I haven't defined a Version field, but the entities that I'm trying to
persist concurrently represents different records.


-----
Not everything that can be counted counts, and not everything that counts can
be counted - Albert Einstein
-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3378805.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

Re: Persist issue in multithreaded environment

Posted by Rick Curtis <cu...@gmail.com>.
Claudio -

This is a bit of a stretch, but do you have a Version field [1] on each of
your Entities?

- Rick

[1]
http://openjpa.apache.org/builds/latest/docs/manual/manual.html#jpa_overview_pc_version


Claudio Di Vita wrote:
> 
> Hi to all,
> 
> I'm having some troubles using OpenJPA in a multithreaded environment.
> 
> Sometimes (because it happens randomly) some entities aren't persisted,
> but OpenJPA doesn't throw any exception. Please note that I persist a set
> of entities (instances of the same class) that don't share any kind of
> information.
> 
> I haven't set the property openjpa.Multithreaded to true because I use a
> different EntityManager for each thread (I get it through a ThreadLocal).
> Obviously If I set this property everything seems to work right.
> 
> So I want to understand why I have to set this property: I'm not doing
> something right or property openjpa.Multithreaded is mandatory for
> multithreaded environments (even using different EntityManagers) ??
> 
> Thank you for the help,
> 
> Claudio
> 

-- 
View this message in context: http://n2.nabble.com/Persist-issue-in-multithreaded-environment-tp3377510p3378077.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.