You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Aron Lurie <ar...@cambridgesemantics.com> on 2011/06/14 16:45:18 UTC
Speeding up commit
Hello,
I have a situation where I am trying to persist ~2500 objects all at
once. After the objects are persisted, the program ends, so there is no
need for any functionality after the objects have been persisted. I am
just trying to dump these objects to the DB as fast as possible.
Using Eclipse's TPTP profiler, I have timed the EntityManager.commit()
operation to take 586 seconds. Of that, cumulatively almost 455 seconds
are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool,
bool). From the point that I start commit(), those 2nd class objects are
not mutated by my program, and I do not need to use any of these objects
after commit() finishes. From what I have read, it would seem that I
have no need for my objects to be proxied, and from my tests, it seems
that removing proxies would speed things up significantly. Am I missing
something? Or is there a way to turn off proxies? And if not, how can I
speed up the proxying operation?
Thanks,
Aron
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
Yes proxy could track changes in second-class objects, if you tell them to do
so.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479464.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
Looks like more than simple bulk insertion to me :)
The interesting way you have chosen should not make first-class managed
object be proxied. That is my remote observation. However, you can inspect
that actual runtime class after an instance has entered a persistence
context, to see for yourself.
A a = new A(); B b = new B();
a.setB(b);
em.persist(a); em.persist(b);
assertSame(B.class, a.getB().getClass());
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6480642.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
> Adding another property to
> the ProxyManager to not use proxies from the get-go makes sense to me.
Actually, I have considered that option. The problem with it is that
ProxyManager is per unit, not per context. Going via the route that I have
taken allows the user to deactivate proxy for one context (say when it is
loading data in bulk) without affecting behavior for other contexts.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479527.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
@Mike,
OPENJPA-2017
The use case is important and useful.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479476.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
Aron -
Can you post your Entities?
Re: Speeding up commit
Posted by Aron Lurie <ar...@cambridgesemantics.com>.
I am enhancing during runtime, but with the build-time enhancer. I know
that sounds kind of strange, but I am using PCEnhance.run and a custom
BytecodeWriter that writes to a custom Class Loader that allows me to
add and use the enhanced classes during runtime.
Would this cause managed objects to be proxied? Would it matter if the
declared return type is different from the type I specify as targetEntity?
-Aron
On 6/15/2011 4:43 PM, Pinaki Poddar wrote:
> Proxies are *not* created for managed objects, in general. This brings me to
> my favorite question:
> are you enhancing the persistent classes at build-time? If not, please do.
> It will save you ton of trouble.
>
> For normal (I mean build-time enhanced) persistence mode, OpenJPA creates
> proxies for 'second-class objects' -- the objects that are "managed" by the
> runtime in some sense but *could* mutate without OpenJPA knowing about such
> mutation. Examples are instance of java.util.Date or java.util.Collection.
> OpenJPA proxies them because then it can be aware of their mutation and act
> appropriately to capture these changes.
>
>
>
> -----
> Pinaki
> --
> View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6480502.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
Proxies are *not* created for managed objects, in general. This brings me to
my favorite question:
are you enhancing the persistent classes at build-time? If not, please do.
It will save you ton of trouble.
For normal (I mean build-time enhanced) persistence mode, OpenJPA creates
proxies for 'second-class objects' -- the objects that are "managed" by the
runtime in some sense but *could* mutate without OpenJPA knowing about such
mutation. Examples are instance of java.util.Date or java.util.Collection.
OpenJPA proxies them because then it can be aware of their mutation and act
appropriately to capture these changes.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6480502.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
RE: Speeding up commit
Posted by C N Davies <cn...@cndavies.com>.
Actually on that point I think I agree that I have seen this very behaviour
myself. OpenJPA gets confused if change B then change A then commit A (with
cascade all set on A), I commonly get detached entity errors in reference to
B. I ended up working around that by persisting B, reread it then
persisting A. I got so tired of this issue I have been using Hibernate a lot
more lately.
Chris
-----Original Message-----
From: Aron Lurie [mailto:aron@cambridgesemantics.com]
Sent: Thursday, 16 June 2011 5:44 AM
To: users@openjpa.apache.org
Subject: Re: Speeding up commit
Also,
When class A has a property of type class B, and both instances are
persisted, the B instance does not need to be proxied because any mutation
made on the B instance will be noticed by the entity manager since it is
persisted. However, I believe the current behavior is to create a proxy for
the B instance. If I am right, this is a source of redundancy that could be
eliminated.
-Aron
On 6/15/2011 3:11 PM, Aron Lurie wrote:
> Based on the documentation I've read, it didn't occur to me as a user
> that proxying stemmed from detachment. I had the concept in mind that
> proxying was a two part process: attachment - or putting the proxy in
> place of the original object, and detachment - or replacing the
> original object.
>
> Just my 2 cents.
>
> Anyways, I've found a way to speed up the proxying process, by caching
> the result of my objects getters, since they take time to produce the
> return value.
>
> -Aron
>
> On 6/15/2011 11:47 AM, Pinaki Poddar wrote:
>>> it still attaches proxies during commit,
>> It was my bad. Looks like your environment is ready to absorb changes.
>> Please update on trunk -- that has a correct NONE logic that will
>> bypass proxy on commit.
>>
>> -----
>> Pinaki
>> --
>> View this message in context:
>> http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479
>> 404.html Sent from the OpenJPA Users mailing list archive at
>> Nabble.com.
>
Re: Speeding up commit
Posted by Aron Lurie <ar...@cambridgesemantics.com>.
Also,
When class A has a property of type class B, and both instances are
persisted, the B instance does not need to be proxied because any
mutation made on the B instance will be noticed by the entity manager
since it is persisted. However, I believe the current behavior is to
create a proxy for the B instance. If I am right, this is a source of
redundancy that could be eliminated.
-Aron
On 6/15/2011 3:11 PM, Aron Lurie wrote:
> Based on the documentation I've read, it didn't occur to me as a user
> that proxying stemmed from detachment. I had the concept in mind that
> proxying was a two part process: attachment - or putting the proxy in
> place of the original object, and detachment - or replacing the
> original object.
>
> Just my 2 cents.
>
> Anyways, I've found a way to speed up the proxying process, by caching
> the result of my objects getters, since they take time to produce the
> return value.
>
> -Aron
>
> On 6/15/2011 11:47 AM, Pinaki Poddar wrote:
>>> it still attaches proxies during commit,
>> It was my bad. Looks like your environment is ready to absorb changes.
>> Please update on trunk -- that has a correct NONE logic that will bypass
>> proxy on commit.
>>
>> -----
>> Pinaki
>> --
>> View this message in context:
>> http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479404.html
>> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>
Re: Speeding up commit
Posted by Aron Lurie <ar...@cambridgesemantics.com>.
Based on the documentation I've read, it didn't occur to me as a user
that proxying stemmed from detachment. I had the concept in mind that
proxying was a two part process: attachment - or putting the proxy in
place of the original object, and detachment - or replacing the original
object.
Just my 2 cents.
Anyways, I've found a way to speed up the proxying process, by caching
the result of my objects getters, since they take time to produce the
return value.
-Aron
On 6/15/2011 11:47 AM, Pinaki Poddar wrote:
>> it still attaches proxies during commit,
> It was my bad. Looks like your environment is ready to absorb changes.
> Please update on trunk -- that has a correct NONE logic that will bypass
> proxy on commit.
>
> -----
> Pinaki
> --
> View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479404.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
> it still attaches proxies during commit,
It was my bad. Looks like your environment is ready to absorb changes.
Please update on trunk -- that has a correct NONE logic that will bypass
proxy on commit.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479404.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
There was an error in initial commit. Please try Revision 1135857.
Also please let us know your observation with this changes.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479537.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Michael Dick <mi...@gmail.com>.
Unfortunately we don't have a good answer for your use case. I spent a
little (very little) time looking into removing the proxies altogether and
it wasn't as simple as just putting the non proxies into the entity.
There should be a jira created for this use case, but I haven't been able to
find one. Do you have a JIRA acount, and would you mind opening one?
-mike
On Wed, Jun 15, 2011 at 9:52 AM, Aron Lurie <ar...@cambridgesemantics.com>wrote:
> We also need the proxies for for managed entities to track the state of
>> non
>> primitives (Dates, collections etc.).
>>
> So in my application, I know that the state of the non primitives will not
> be changing. All I need OpenJPA to do is extract the value of the
> non-primitive once when it is creating its insert statement.
>
>
> It will not proxy with for AutoDetachType.NONE.
>>
> I have already set this property, and it allows me to close the entity
> manager much faster because it does not bother detaching the proxies, but it
> still attaches proxies during commit, so commit does not run any faster.
>
> -Aron
>
>
> On 6/15/2011 10:37 AM, Michael Dick wrote:
>
>> Rick's right on both counts. TrackChanges doesn't eliminate proxies - it
>> should just make them no-op.
>>
>> We also need the proxies for for managed entities to track the state of
>> non
>> primitives (Dates, collections etc.). I don't think we have code in place
>> that falls back and does a more thorough comparison if the proxies are not
>> found though.
>>
>> Pinaki,
>>
>> The code changes are definitely untested - it's currently breaks the
>> TestEnumToKernelConstantMappings test (which is rather banal, but probably
>> there for a good reason).
>>
>> I'm not sure what you mean about not having a regression test environment.
>> This problem would be found in a rather quick maven build. I understand
>> not
>> having multiple databases available, but running the regression bucket
>> with
>> derby should be doable.
>>
>> You can skip the long running locking tests with this arg:
>> -Dsurefire.excludes.locking=**, if time is a concern.
>>
>> -mike
>>
>> On Wed, Jun 15, 2011 at 9:11 AM, Rick Curtis<cu...@gmail.com> wrote:
>>
>> Javadoc from ProxyManagerImpl
>>>
>>> /**
>>> * Whether proxies produced by this factory will use {@link
>>> ChangeTracker}s
>>> * to try to cut down on data store operations at the cost of some
>>> extra
>>> * bookkeeping overhead. Defaults to true.
>>> */
>>> public boolean getTrackChanges() {
>>> return _trackChanges;
>>> }
>>>
>>> It sounds like this property is used to determine whether the proxies are
>>> tracking changes... not to toggle the creation. Adding another property
>>> to
>>> the ProxyManager to not use proxies from the get-go makes sense to me.
>>>
>>> On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter<kw...@gmail.com>
>>> wrote:
>>>
>>> Hi guys,
>>>> Shouldn't this property setting turn off the proxy usage?
>>>>
>>>> <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
>>>>
>>>> That's the way I read the documentation, but it doesn't seem to work
>>>> that
>>>> way. We still get the proxies created. Actually, I don't see much
>>>> difference in processing whether this is set to True or False. Is this
>>>> a
>>>> bug, or am I reading the documentation wrong?
>>>>
>>>> Thanks,
>>>> Kevin
>>>>
>>>> On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis<cu...@gmail.com>
>>>> wrote:
>>>>
>>>> Aren't proxies also used to track changes while a persistence context
>>>>>
>>>> is
>>>
>>>> active?
>>>>>
>>>>> Rick Curtis
>>>>>
>>>>>
>>>
>>> --
>>> *Rick Curtis*
>>>
>>>
>
Re: Speeding up commit
Posted by Aron Lurie <ar...@cambridgesemantics.com>.
> We also need the proxies for for managed entities to track the state of non
> primitives (Dates, collections etc.).
So in my application, I know that the state of the non primitives will
not be changing. All I need OpenJPA to do is extract the value of the
non-primitive once when it is creating its insert statement.
> It will not proxy with for AutoDetachType.NONE.
I have already set this property, and it allows me to close the entity
manager much faster because it does not bother detaching the proxies,
but it still attaches proxies during commit, so commit does not run any
faster.
-Aron
On 6/15/2011 10:37 AM, Michael Dick wrote:
> Rick's right on both counts. TrackChanges doesn't eliminate proxies - it
> should just make them no-op.
>
> We also need the proxies for for managed entities to track the state of non
> primitives (Dates, collections etc.). I don't think we have code in place
> that falls back and does a more thorough comparison if the proxies are not
> found though.
>
> Pinaki,
>
> The code changes are definitely untested - it's currently breaks the
> TestEnumToKernelConstantMappings test (which is rather banal, but probably
> there for a good reason).
>
> I'm not sure what you mean about not having a regression test environment.
> This problem would be found in a rather quick maven build. I understand not
> having multiple databases available, but running the regression bucket with
> derby should be doable.
>
> You can skip the long running locking tests with this arg:
> -Dsurefire.excludes.locking=**, if time is a concern.
>
> -mike
>
> On Wed, Jun 15, 2011 at 9:11 AM, Rick Curtis<cu...@gmail.com> wrote:
>
>> Javadoc from ProxyManagerImpl
>>
>> /**
>> * Whether proxies produced by this factory will use {@link
>> ChangeTracker}s
>> * to try to cut down on data store operations at the cost of some extra
>> * bookkeeping overhead. Defaults to true.
>> */
>> public boolean getTrackChanges() {
>> return _trackChanges;
>> }
>>
>> It sounds like this property is used to determine whether the proxies are
>> tracking changes... not to toggle the creation. Adding another property to
>> the ProxyManager to not use proxies from the get-go makes sense to me.
>>
>> On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter<kw...@gmail.com> wrote:
>>
>>> Hi guys,
>>> Shouldn't this property setting turn off the proxy usage?
>>>
>>> <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
>>>
>>> That's the way I read the documentation, but it doesn't seem to work that
>>> way. We still get the proxies created. Actually, I don't see much
>>> difference in processing whether this is set to True or False. Is this a
>>> bug, or am I reading the documentation wrong?
>>>
>>> Thanks,
>>> Kevin
>>>
>>> On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis<cu...@gmail.com> wrote:
>>>
>>>> Aren't proxies also used to track changes while a persistence context
>> is
>>>> active?
>>>>
>>>> Rick Curtis
>>>>
>>
>>
>> --
>> *Rick Curtis*
>>
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
@Mike
will get it sorted. Please allow some time. I am out of practice and my old
laptop is gone :)
I think Hudson build is quiet now -- last time it was unhappy was about
7:18PM night before.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479506.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Michael Dick <mi...@gmail.com>.
Rick's right on both counts. TrackChanges doesn't eliminate proxies - it
should just make them no-op.
We also need the proxies for for managed entities to track the state of non
primitives (Dates, collections etc.). I don't think we have code in place
that falls back and does a more thorough comparison if the proxies are not
found though.
Pinaki,
The code changes are definitely untested - it's currently breaks the
TestEnumToKernelConstantMappings test (which is rather banal, but probably
there for a good reason).
I'm not sure what you mean about not having a regression test environment.
This problem would be found in a rather quick maven build. I understand not
having multiple databases available, but running the regression bucket with
derby should be doable.
You can skip the long running locking tests with this arg:
-Dsurefire.excludes.locking=**, if time is a concern.
-mike
On Wed, Jun 15, 2011 at 9:11 AM, Rick Curtis <cu...@gmail.com> wrote:
> Javadoc from ProxyManagerImpl
>
> /**
> * Whether proxies produced by this factory will use {@link
> ChangeTracker}s
> * to try to cut down on data store operations at the cost of some extra
> * bookkeeping overhead. Defaults to true.
> */
> public boolean getTrackChanges() {
> return _trackChanges;
> }
>
> It sounds like this property is used to determine whether the proxies are
> tracking changes... not to toggle the creation. Adding another property to
> the ProxyManager to not use proxies from the get-go makes sense to me.
>
> On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter <kw...@gmail.com> wrote:
>
> > Hi guys,
> > Shouldn't this property setting turn off the proxy usage?
> >
> > <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
> >
> > That's the way I read the documentation, but it doesn't seem to work that
> > way. We still get the proxies created. Actually, I don't see much
> > difference in processing whether this is set to True or False. Is this a
> > bug, or am I reading the documentation wrong?
> >
> > Thanks,
> > Kevin
> >
> > On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis <cu...@gmail.com> wrote:
> >
> > > Aren't proxies also used to track changes while a persistence context
> is
> > > active?
> > >
> > > Rick Curtis
> > >
> >
>
>
>
> --
> *Rick Curtis*
>
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
Javadoc from ProxyManagerImpl
/**
* Whether proxies produced by this factory will use {@link
ChangeTracker}s
* to try to cut down on data store operations at the cost of some extra
* bookkeeping overhead. Defaults to true.
*/
public boolean getTrackChanges() {
return _trackChanges;
}
It sounds like this property is used to determine whether the proxies are
tracking changes... not to toggle the creation. Adding another property to
the ProxyManager to not use proxies from the get-go makes sense to me.
On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter <kw...@gmail.com> wrote:
> Hi guys,
> Shouldn't this property setting turn off the proxy usage?
>
> <property name="openjpa.ProxyManager" value="TrackChanges=false"/>
>
> That's the way I read the documentation, but it doesn't seem to work that
> way. We still get the proxies created. Actually, I don't see much
> difference in processing whether this is set to True or False. Is this a
> bug, or am I reading the documentation wrong?
>
> Thanks,
> Kevin
>
> On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis <cu...@gmail.com> wrote:
>
> > Aren't proxies also used to track changes while a persistence context is
> > active?
> >
> > Rick Curtis
> >
>
--
*Rick Curtis*
Re: Speeding up commit
Posted by Kevin Sutter <kw...@gmail.com>.
Hi guys,
Shouldn't this property setting turn off the proxy usage?
<property name="openjpa.ProxyManager" value="TrackChanges=false"/>
That's the way I read the documentation, but it doesn't seem to work that
way. We still get the proxies created. Actually, I don't see much
difference in processing whether this is set to True or False. Is this a
bug, or am I reading the documentation wrong?
Thanks,
Kevin
On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis <cu...@gmail.com> wrote:
> Aren't proxies also used to track changes while a persistence context is
> active?
>
> Rick Curtis
>
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
Aren't proxies also used to track changes while a persistence context is
active?
Rick Curtis
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
Proxying/Deproxying stems from detachment. We proxy because we want to tract
changes in detached objects. AutoDetach.NONE basically tells: 'do not bother
detaching at all, not going to use these objects anyway ever'.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6476649.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
> It will not proxy with for AutoDetachType.NONE.
That is confusing my simple mind.... Why do I need to set an AutoDetach flag
to tell OpenJPA to not use proxies?
>These stuff is highly untested because I do not have access to a ...
I'm not hung up on the right/wrong-ness of your change... I'm still thinking
about it conceptually.
On Tue, Jun 14, 2011 at 4:43 PM, Pinaki Poddar <pp...@apache.org> wrote:
> It will not proxy with for AutoDetachType.NONE.
> See StateManagerImpl commit log for further details.
>
> These stuff is highly untested because I do not have access to a regression
> test environment right now. I am still committing so that it gets tested
> with other commits. I will rollback if things just start breaking.
>
> -----
> Pinaki
>
> --
*Rick Curtis*
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
It will not proxy with for AutoDetachType.NONE.
See StateManagerImpl commit log for further details.
These stuff is highly untested because I do not have access to a regression
test environment right now. I am still committing so that it gets tested
with other commits. I will rollback if things just start breaking.
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6476127.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
Is the problem when OpenJPA tries to proxy user supplied objects on
transaction commit(flush)? Or is the problem when the persistence context
detaches and OpenJPA spends cycles trying to remove the proxies that we just
put in?
Pinaki - Did you solve the both problems, or just the second? ... I thought
the user Aron was reporting a problem with the fact that we ever put proxies
in.?
On Tue, Jun 14, 2011 at 3:32 PM, Pinaki Poddar <pp...@apache.org> wrote:
> Hello,
> For batch insertion,
> 1. Set openjpa.RetainState=false
> this might gain few extra cycles
>
> I had introduced a tentative option [1] for automatic detachment (your
> observation of cycles being spent on StateManager.proxyFields(...) is
> related to detachment process that OpenJPA runtime invokes at
> user-configurable points in the lifetime of a persistence context such as
> transaction endpoints or when EntityManager.close() is called).
>
> If the user application does not intend to refer to the managed entities
> later (as it seems to fits the use case you are describing), then we can
> bypass the overhead of detachment.
>
> In a quick Person-has-many-Address type model, I notice a ~20% reduction
> in data insert with this new NONE option.
>
> To access, and if possible verify efficacy of this new feature, please do
> the following
>
> 1. get a nightly build or build from trunk locally.
> 2. Ensure that revision 1135776 is included. Easy way to do this
> $ java -jar openjpa.jar
> will print the latest revision included in the jar
> 3. In user application,
> em.setProperty("openjpa.AutoDetach", AutoDetachType.NONE);
> Note that the option must be set on an instantiated EntityManager, and
> *not* in persistence.xml
>
> [1] https://issues.apache.org/jira/browse/OPENJPA-2017
>
> -----
> Pinaki
>
> --
*Rick Curtis*
Re: Speeding up commit
Posted by Pinaki Poddar <pp...@apache.org>.
Hello,
For batch insertion,
1. Set openjpa.RetainState=false
this might gain few extra cycles
I had introduced a tentative option [1] for automatic detachment (your
observation of cycles being spent on StateManager.proxyFields(...) is
related to detachment process that OpenJPA runtime invokes at
user-configurable points in the lifetime of a persistence context such as
transaction endpoints or when EntityManager.close() is called).
If the user application does not intend to refer to the managed entities
later (as it seems to fits the use case you are describing), then we can
bypass the overhead of detachment.
In a quick Person-has-many-Address type model, I notice a ~20% reduction
in data insert with this new NONE option.
To access, and if possible verify efficacy of this new feature, please do
the following
1. get a nightly build or build from trunk locally.
2. Ensure that revision 1135776 is included. Easy way to do this
$ java -jar openjpa.jar
will print the latest revision included in the jar
3. In user application,
em.setProperty("openjpa.AutoDetach", AutoDetachType.NONE);
Note that the option must be set on an instantiated EntityManager, and
*not* in persistence.xml
[1] https://issues.apache.org/jira/browse/OPENJPA-2017
-----
Pinaki
--
View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6475817.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
Unfortunately it's looking like a this is a code path which we don't have an
optimizations for.... Essentially you're using OpenJPA to put data into a DB
and then once your tran commits you're not going to use the Entities any
longer.? If you're looking for a workaround, you could noop
org.apache.openjpa.kernel.SingleFieldManager.proxy(...) and just return
true. Not ideal, but it might work for you in the short term.
On Tue, Jun 14, 2011 at 10:35 AM, C N Davies <cn...@cndavies.com> wrote:
> Hi Aron,
>
> Yes that is what I meant, I persist and commit 200 at a time. When my
> objects are highly complex I threw in em.clear() as I was getting stack
> issues as well.
>
> Chris
>
> -----Original Message-----
> From: Aron Lurie [mailto:aron@cambridgesemantics.com]
> Sent: Wednesday, 15 June 2011 1:06 AM
> To: users@openjpa.apache.org
> Subject: Re: Speeding up commit
>
> Chris,
> What do you mean by batching? Do you mean persisting 200, committing,
> persisting another 200, committing, etc? If so, I don't have the option of
> doing this because my objects reference one another so the cascading
> persist
> would blow up my batch size.
>
> Thanks,
> Aron
>
> On 6/14/2011 10:54 AM, C N Davies wrote:
> > I had a load of issues with committing large quantities of entities
> > (OpenJPA 2.0), in the end I batched them into batches of 200 and the
> > overall performance was significantly better. This is not a
> > scientific analysis of the issue I know, but might be of use to you :)
> >
> > Chris
> >
> >
> > -----Original Message-----
> > From: Aron Lurie [mailto:aron@cambridgesemantics.com]
> > Sent: Wednesday, 15 June 2011 12:50 AM
> > To: users@openjpa.apache.org
> > Subject: Re: Speeding up commit
> >
> > 2.1.0
> >
> > On 6/14/2011 10:47 AM, Rick Curtis wrote:
> >> What version of OpenJPA are you running?
> >>
> >> On Tue, Jun 14, 2011 at 9:45 AM, Aron
> > Lurie<ar...@cambridgesemantics.com>wrote:
> >>> Hello,
> >>> I have a situation where I am trying to persist ~2500 objects all at
> > once.
> >>> After the objects are persisted, the program ends, so there is no
> >>> need for any functionality after the objects have been persisted. I
> >>> am just trying to dump these objects to the DB as fast as possible.
> >>>
> >>> Using Eclipse's TPTP profiler, I have timed the
> >>> EntityManager.commit() operation to take 586 seconds. Of that,
> >>> cumulatively almost 455 seconds are spent inside of ~2500 calls to
> > StateManagerImpl.proxyFields(bool, bool).
> >>> From the point that I start commit(), those 2nd class objects are
> >>> not mutated by my program, and I do not need to use any of these
> >>> objects after
> >>> commit() finishes. From what I have read, it would seem that I have
> >>> no need for my objects to be proxied, and from my tests, it seems
> >>> that removing proxies would speed things up significantly. Am I
> >>> missing something? Or is there a way to turn off proxies? And if
> >>> not, how can I speed up the proxying operation?
> >>>
> >>> Thanks,
> >>> Aron
> >>>
> >>
>
>
--
*Rick Curtis*
RE: Speeding up commit
Posted by C N Davies <cn...@cndavies.com>.
Hi Aron,
Yes that is what I meant, I persist and commit 200 at a time. When my
objects are highly complex I threw in em.clear() as I was getting stack
issues as well.
Chris
-----Original Message-----
From: Aron Lurie [mailto:aron@cambridgesemantics.com]
Sent: Wednesday, 15 June 2011 1:06 AM
To: users@openjpa.apache.org
Subject: Re: Speeding up commit
Chris,
What do you mean by batching? Do you mean persisting 200, committing,
persisting another 200, committing, etc? If so, I don't have the option of
doing this because my objects reference one another so the cascading persist
would blow up my batch size.
Thanks,
Aron
On 6/14/2011 10:54 AM, C N Davies wrote:
> I had a load of issues with committing large quantities of entities
> (OpenJPA 2.0), in the end I batched them into batches of 200 and the
> overall performance was significantly better. This is not a
> scientific analysis of the issue I know, but might be of use to you :)
>
> Chris
>
>
> -----Original Message-----
> From: Aron Lurie [mailto:aron@cambridgesemantics.com]
> Sent: Wednesday, 15 June 2011 12:50 AM
> To: users@openjpa.apache.org
> Subject: Re: Speeding up commit
>
> 2.1.0
>
> On 6/14/2011 10:47 AM, Rick Curtis wrote:
>> What version of OpenJPA are you running?
>>
>> On Tue, Jun 14, 2011 at 9:45 AM, Aron
> Lurie<ar...@cambridgesemantics.com>wrote:
>>> Hello,
>>> I have a situation where I am trying to persist ~2500 objects all at
> once.
>>> After the objects are persisted, the program ends, so there is no
>>> need for any functionality after the objects have been persisted. I
>>> am just trying to dump these objects to the DB as fast as possible.
>>>
>>> Using Eclipse's TPTP profiler, I have timed the
>>> EntityManager.commit() operation to take 586 seconds. Of that,
>>> cumulatively almost 455 seconds are spent inside of ~2500 calls to
> StateManagerImpl.proxyFields(bool, bool).
>>> From the point that I start commit(), those 2nd class objects are
>>> not mutated by my program, and I do not need to use any of these
>>> objects after
>>> commit() finishes. From what I have read, it would seem that I have
>>> no need for my objects to be proxied, and from my tests, it seems
>>> that removing proxies would speed things up significantly. Am I
>>> missing something? Or is there a way to turn off proxies? And if
>>> not, how can I speed up the proxying operation?
>>>
>>> Thanks,
>>> Aron
>>>
>>
Re: Speeding up commit
Posted by Aron Lurie <ar...@cambridgesemantics.com>.
Chris,
What do you mean by batching? Do you mean persisting 200, committing,
persisting another 200, committing, etc? If so, I don't have the option
of doing this because my objects reference one another so the cascading
persist would blow up my batch size.
Thanks,
Aron
On 6/14/2011 10:54 AM, C N Davies wrote:
> I had a load of issues with committing large quantities of entities (OpenJPA
> 2.0), in the end I batched them into batches of 200 and the overall
> performance was significantly better. This is not a scientific analysis of
> the issue I know, but might be of use to you :)
>
> Chris
>
>
> -----Original Message-----
> From: Aron Lurie [mailto:aron@cambridgesemantics.com]
> Sent: Wednesday, 15 June 2011 12:50 AM
> To: users@openjpa.apache.org
> Subject: Re: Speeding up commit
>
> 2.1.0
>
> On 6/14/2011 10:47 AM, Rick Curtis wrote:
>> What version of OpenJPA are you running?
>>
>> On Tue, Jun 14, 2011 at 9:45 AM, Aron
> Lurie<ar...@cambridgesemantics.com>wrote:
>>> Hello,
>>> I have a situation where I am trying to persist ~2500 objects all at
> once.
>>> After the objects are persisted, the program ends, so there is no
>>> need for any functionality after the objects have been persisted. I
>>> am just trying to dump these objects to the DB as fast as possible.
>>>
>>> Using Eclipse's TPTP profiler, I have timed the
>>> EntityManager.commit() operation to take 586 seconds. Of that,
>>> cumulatively almost 455 seconds are spent inside of ~2500 calls to
> StateManagerImpl.proxyFields(bool, bool).
>>> From the point that I start commit(), those 2nd class objects are
>>> not mutated by my program, and I do not need to use any of these
>>> objects after
>>> commit() finishes. From what I have read, it would seem that I have
>>> no need for my objects to be proxied, and from my tests, it seems
>>> that removing proxies would speed things up significantly. Am I
>>> missing something? Or is there a way to turn off proxies? And if not,
>>> how can I speed up the proxying operation?
>>>
>>> Thanks,
>>> Aron
>>>
>>
RE: Speeding up commit
Posted by C N Davies <cn...@cndavies.com>.
I had a load of issues with committing large quantities of entities (OpenJPA
2.0), in the end I batched them into batches of 200 and the overall
performance was significantly better. This is not a scientific analysis of
the issue I know, but might be of use to you :)
Chris
-----Original Message-----
From: Aron Lurie [mailto:aron@cambridgesemantics.com]
Sent: Wednesday, 15 June 2011 12:50 AM
To: users@openjpa.apache.org
Subject: Re: Speeding up commit
2.1.0
On 6/14/2011 10:47 AM, Rick Curtis wrote:
> What version of OpenJPA are you running?
>
> On Tue, Jun 14, 2011 at 9:45 AM, Aron
Lurie<ar...@cambridgesemantics.com>wrote:
>
>> Hello,
>> I have a situation where I am trying to persist ~2500 objects all at
once.
>> After the objects are persisted, the program ends, so there is no
>> need for any functionality after the objects have been persisted. I
>> am just trying to dump these objects to the DB as fast as possible.
>>
>> Using Eclipse's TPTP profiler, I have timed the
>> EntityManager.commit() operation to take 586 seconds. Of that,
>> cumulatively almost 455 seconds are spent inside of ~2500 calls to
StateManagerImpl.proxyFields(bool, bool).
>> From the point that I start commit(), those 2nd class objects are
>> not mutated by my program, and I do not need to use any of these
>> objects after
>> commit() finishes. From what I have read, it would seem that I have
>> no need for my objects to be proxied, and from my tests, it seems
>> that removing proxies would speed things up significantly. Am I
>> missing something? Or is there a way to turn off proxies? And if not,
>> how can I speed up the proxying operation?
>>
>> Thanks,
>> Aron
>>
>
>
Re: Speeding up commit
Posted by Aron Lurie <ar...@cambridgesemantics.com>.
2.1.0
On 6/14/2011 10:47 AM, Rick Curtis wrote:
> What version of OpenJPA are you running?
>
> On Tue, Jun 14, 2011 at 9:45 AM, Aron Lurie<ar...@cambridgesemantics.com>wrote:
>
>> Hello,
>> I have a situation where I am trying to persist ~2500 objects all at once.
>> After the objects are persisted, the program ends, so there is no need for
>> any functionality after the objects have been persisted. I am just trying to
>> dump these objects to the DB as fast as possible.
>>
>> Using Eclipse's TPTP profiler, I have timed the EntityManager.commit()
>> operation to take 586 seconds. Of that, cumulatively almost 455 seconds are
>> spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool).
>> From the point that I start commit(), those 2nd class objects are not
>> mutated by my program, and I do not need to use any of these objects after
>> commit() finishes. From what I have read, it would seem that I have no need
>> for my objects to be proxied, and from my tests, it seems that removing
>> proxies would speed things up significantly. Am I missing something? Or is
>> there a way to turn off proxies? And if not, how can I speed up the proxying
>> operation?
>>
>> Thanks,
>> Aron
>>
>
>
Re: Speeding up commit
Posted by Rick Curtis <cu...@gmail.com>.
What version of OpenJPA are you running?
On Tue, Jun 14, 2011 at 9:45 AM, Aron Lurie <ar...@cambridgesemantics.com>wrote:
> Hello,
> I have a situation where I am trying to persist ~2500 objects all at once.
> After the objects are persisted, the program ends, so there is no need for
> any functionality after the objects have been persisted. I am just trying to
> dump these objects to the DB as fast as possible.
>
> Using Eclipse's TPTP profiler, I have timed the EntityManager.commit()
> operation to take 586 seconds. Of that, cumulatively almost 455 seconds are
> spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool).
> From the point that I start commit(), those 2nd class objects are not
> mutated by my program, and I do not need to use any of these objects after
> commit() finishes. From what I have read, it would seem that I have no need
> for my objects to be proxied, and from my tests, it seems that removing
> proxies would speed things up significantly. Am I missing something? Or is
> there a way to turn off proxies? And if not, how can I speed up the proxying
> operation?
>
> Thanks,
> Aron
>
--
*Rick Curtis*