You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@syncope.apache.org by Aniket Band <ab...@talend.com> on 2016/03/16 16:27:58 UTC
Deadlocks - Performance Issue
Hello All,
One of our customers uses Syncope (1.2.5) and got some performance issues. They want to upload 100.000 users via REST interface. They created 3-5 routes in parallel to call the rest interface. Unfortunately it requires 2 day to insert 100.000 users. As far as I know it should not take 2 days. The Log claims some deadlocks in the database.
Any idea about it? In the mailing list and in JIRA I could not find any similar issue.
Database is a MSSQL in a cluster. Thanks in advance for your inputs.
Thanks,
Aniket
Re: Deadlocks - Performance Issue
Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks Guido! We will experiment with increasing "maxActive". Here is an
example of where the deadlock occurs:
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Transaction
(Process ID 108) was deadlocked on lock resources with another process and
has been chosen as the deadlock victim. Rerun the transact
ion. {prepstmnt 195680672 UPDATE dbo.Membership SET lastChangeDate = ?,
SYNCOPEUSER_ID = ? WHERE id = ?} [code=1205, state=40001]
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:219)
~[openjpa-lib-2.3.0.jar:2.3.0]
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:195)
~[openjpa-lib-2.3.0.jar:2.3.0]
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$1000(LoggingConnectionDecorator.java:59)
~[openjpa-lib-2.3.0.jar:2.3.0]
--
On Mon, Mar 21, 2016 at 12:29 PM, Guido Wimmel <gu...@gmx.net> wrote:
> Hi,
>
> no idea if this could be helpful, but I could reproduce a deadlock some
> time ago in Syncope by an artificial load test with many parallel threads.
> The deadlock occurred when there were no more connections available in the
> connection pool.
> You could check if the connection pool (property jpa.pool.maxActive) is
> sufficiently large.
>
> The mentioned log that "claims some deadlocks" could also be helpful to
> see where the deadlock exactly happens.
>
> Cheers,
> Guido
>
> Am 18.03.2016 um 17:56 schrieb Colm O hEigeartaigh:
>
> Yes, the users are stored internally after creation. I guess we will have
> to set up a test-case and try to reproduce the issue.
>
> Thanks for your help Giacomo!
>
> Colm.
>
> On Fri, Mar 18, 2016 at 11:08 AM, Giacomo Lamonaco <
> <gi...@tirasa.net> wrote:
>
>> Hi Colm,
>>
>> I tried to inspect the latest fixes but I didn't find something related
>> to performance or deadlock.
>>
>> I have some more questions:
>> 1) At the end of the creation process, users have been stored internally?
>> 2) Could you provide the average time required for an user creation? And
>> for an user search?
>>
>> It could be better to test 2) both with and without the cluster.
>>
>> Anyway, my suggestion is to upgrade Syncope to 1.2.8-SNAPSHOT version and
>> start again with the import.
>>
>> Best Regards,
>>
>> --
>> Giacomo Lamonaco
>>
>> Tirasa - Open Source Excellencehttp://www.tirasa.net/
>>
>> Syncope committerhttp://people.apache.org/~giacomolm
>>
>> Il giorno ven, 18/03/2016 alle 09.54 +0000, Colm O hEigeartaigh ha
>> scritto:
>>
>> i Giacomo,
>>
>> My understanding is that there are no external resources involved here.
>> The MSSQL cluster is used for the Syncope internal storage. So the users
>> are created directly in Syncope itself via the REST API. Does this kind of
>> deadlock issue ring any bells for any issue fixed subsequent to 1.2.5? If
>> not we will try to reproduce the issue locally to try to narrow the problem
>> down.
>>
>> Thanks,
>>
>> Colm.
>>
>> On Thu, Mar 17, 2016 at 1:27 PM, Giacomo Lamonaco <
>> <gi...@tirasa.net> wrote:
>>
>> Il giorno mer, 16/03/2016 alle 15.27 +0000, Aniket Band ha scritto:
>> > Hello All,
>> >
>> > One of our customers uses Syncope (1.2.5) and got some performance
>> > issues. They want to upload 100.000 users via REST interface. They
>> > created 3-5 routes in parallel to call the rest interface.
>> > Unfortunately it requires 2 day to insert 100.000 users. As far as I
>> > know it should not take 2 days. The Log claims some deadlocks in the
>> > database.
>> >
>> > Any idea about it? In the mailing list and in JIRA I could not find
>> > any similar issue.
>> >
>> > Database is a MSSQL in a cluster. Thanks in advance for your inputs.
>> >
>> Hi Aniket,
>>
>> Do you have some external resources connected to Syncope? (Connected
>> means that every time you create an user in Syncope, that user is
>> automatically propagated to the external resource).
>>
>> Do you have the possibility to create users just on one MSSQL instance
>> (switching off the cluster)?
>>
>> It's not always a good idea to perform large 'import' using the REST
>> interface. You could try to perform this kind of import at a lower level,
>> maybe creating a dump containing those users.
>>
>> Best Regards,
>>
>> --
>> Giacomo Lamonaco
>>
>> Tirasa - Open Source Excellence
>> http://www.tirasa.net/
>>
>> Syncope committer
>> http://people.apache.org/~giacomolm
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: Deadlocks - Performance Issue
Posted by Guido Wimmel <gu...@gmx.net>.
Hi,
no idea if this could be helpful, but I could reproduce a deadlock some
time ago in Syncope by an artificial load test with many parallel
threads. The deadlock occurred when there were no more connections
available in the connection pool.
You could check if the connection pool (property jpa.pool.maxActive) is
sufficiently large.
The mentioned log that "claims some deadlocks" could also be helpful to
see where the deadlock exactly happens.
Cheers,
Guido
Am 18.03.2016 um 17:56 schrieb Colm O hEigeartaigh:
> Yes, the users are stored internally after creation. I guess we will
> have to set up a test-case and try to reproduce the issue.
>
> Thanks for your help Giacomo!
>
> Colm.
>
> On Fri, Mar 18, 2016 at 11:08 AM, Giacomo Lamonaco
> <giacomo.lamonaco@tirasa.net <ma...@tirasa.net>> wrote:
>
> Hi Colm,
>
> I tried to inspect the latest fixes but I didn't find something
> related to performance or deadlock.
>
> I have some more questions:
> 1) At the end of the creation process, users have been stored
> internally?
> 2) Could you provide the average time required for an user
> creation? And for an user search?
>
> It could be better to test 2) both with and without the cluster.
>
> Anyway, my suggestion is to upgrade Syncope to 1.2.8-SNAPSHOT
> version and start again with the import.
>
> Best Regards,
>
> --
> Giacomo Lamonaco
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Syncope committer
> http://people.apache.org/~giacomolm
> <http://people.apache.org/%7Egiacomolm>
>
> Il giorno ven, 18/03/2016 alle 09.54 +0000, Colm O hEigeartaigh ha
> scritto:
>> i Giacomo,
>>
>> My understanding is that there are no external resources involved
>> here. The MSSQL cluster is used for the Syncope internal storage.
>> So the users are created directly in Syncope itself via the REST
>> API. Does this kind of deadlock issue ring any bells for any
>> issue fixed subsequent to 1.2.5? If not we will try to reproduce
>> the issue locally to try to narrow the problem down.
>>
>> Thanks,
>>
>> Colm.
>>
>> On Thu, Mar 17, 2016 at 1:27 PM, Giacomo Lamonaco
>> <giacomo.lamonaco@tirasa.net
>> <ma...@tirasa.net>> wrote:
>>> Il giorno mer, 16/03/2016 alle 15.27 +0000, Aniket Band ha scritto:
>>> > Hello All,
>>> >
>>> > One of our customers uses Syncope (1.2.5) and got some performance
>>> > issues. They want to upload 100.000 users via REST interface. They
>>> > created 3-5 routes in parallel to call the rest interface.
>>> > Unfortunately it requires 2 day to insert 100.000 users. As
>>> far as I
>>> > know it should not take 2 days. The Log claims some deadlocks
>>> in the
>>> > database.
>>> >
>>> > Any idea about it? In the mailing list and in JIRA I could not
>>> find
>>> > any similar issue.
>>> >
>>> > Database is a MSSQL in a cluster. Thanks in advance for your
>>> inputs.
>>> >
>>> Hi Aniket,
>>>
>>> Do you have some external resources connected to Syncope?
>>> (Connected means that every time you create an user in Syncope,
>>> that user is automatically propagated to the external resource).
>>>
>>> Do you have the possibility to create users just on one MSSQL
>>> instance (switching off the cluster)?
>>>
>>> It's not always a good idea to perform large 'import' using the
>>> REST interface. You could try to perform this kind of import at
>>> a lower level, maybe creating a dump containing those users.
>>>
>>> Best Regards,
>>>
>>> --
>>> Giacomo Lamonaco
>>>
>>> Tirasa - Open Source Excellence
>>> http://www.tirasa.net/
>>>
>>> Syncope committer
>>> http://people.apache.org/~giacomolm
>>> <http://people.apache.org/%7Egiacomolm>
>>>
>>>
>>>
>>
>>
>>
>
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
Re: Deadlocks - Performance Issue
Posted by Colm O hEigeartaigh <co...@apache.org>.
Yes, the users are stored internally after creation. I guess we will have
to set up a test-case and try to reproduce the issue.
Thanks for your help Giacomo!
Colm.
On Fri, Mar 18, 2016 at 11:08 AM, Giacomo Lamonaco <
giacomo.lamonaco@tirasa.net> wrote:
> Hi Colm,
>
> I tried to inspect the latest fixes but I didn't find something related to
> performance or deadlock.
>
> I have some more questions:
> 1) At the end of the creation process, users have been stored internally?
> 2) Could you provide the average time required for an user creation? And
> for an user search?
>
> It could be better to test 2) both with and without the cluster.
>
> Anyway, my suggestion is to upgrade Syncope to 1.2.8-SNAPSHOT version and
> start again with the import.
>
> Best Regards,
>
> --
> Giacomo Lamonaco
>
> Tirasa - Open Source Excellencehttp://www.tirasa.net/
>
> Syncope committerhttp://people.apache.org/~giacomolm
>
>
> Il giorno ven, 18/03/2016 alle 09.54 +0000, Colm O hEigeartaigh ha scritto:
>
> i Giacomo,
>
> My understanding is that there are no external resources involved here.
> The MSSQL cluster is used for the Syncope internal storage. So the users
> are created directly in Syncope itself via the REST API. Does this kind of
> deadlock issue ring any bells for any issue fixed subsequent to 1.2.5? If
> not we will try to reproduce the issue locally to try to narrow the problem
> down.
>
> Thanks,
>
> Colm.
>
> On Thu, Mar 17, 2016 at 1:27 PM, Giacomo Lamonaco <
> giacomo.lamonaco@tirasa.net> wrote:
>
> Il giorno mer, 16/03/2016 alle 15.27 +0000, Aniket Band ha scritto:
> > Hello All,
> >
> > One of our customers uses Syncope (1.2.5) and got some performance
> > issues. They want to upload 100.000 users via REST interface. They
> > created 3-5 routes in parallel to call the rest interface.
> > Unfortunately it requires 2 day to insert 100.000 users. As far as I
> > know it should not take 2 days. The Log claims some deadlocks in the
> > database.
> >
> > Any idea about it? In the mailing list and in JIRA I could not find
> > any similar issue.
> >
> > Database is a MSSQL in a cluster. Thanks in advance for your inputs.
> >
> Hi Aniket,
>
> Do you have some external resources connected to Syncope? (Connected means
> that every time you create an user in Syncope, that user is automatically
> propagated to the external resource).
>
> Do you have the possibility to create users just on one MSSQL instance
> (switching off the cluster)?
>
> It's not always a good idea to perform large 'import' using the REST
> interface. You could try to perform this kind of import at a lower level,
> maybe creating a dump containing those users.
>
> Best Regards,
>
> --
> Giacomo Lamonaco
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Syncope committer
> http://people.apache.org/~giacomolm
>
>
>
>
>
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: Deadlocks - Performance Issue
Posted by Giacomo Lamonaco <gi...@tirasa.net>.
Hi Colm,
I tried to inspect the latest fixes but I didn't find something related
to performance or deadlock.
I have some more questions:
1) At the end of the creation process, users have been stored
internally?
2) Could you provide the average time required for an user creation?
And for an user search?
It could be better to test 2) both with and without the cluster.
Anyway, my suggestion is to upgrade Syncope to 1.2.8-SNAPSHOT version
and start again with the import.
Best Regards,
--
Giacomo Lamonaco
Tirasa - Open Source Excellence
http://www.tirasa.net/
Syncope committer
http://people.apache.org/~giacomolm
Il giorno ven, 18/03/2016 alle 09.54 +0000, Colm O hEigeartaigh ha
scritto:
> i Giacomo,
>
> My understanding is that there are no external resources involved
> here. The MSSQL cluster is used for the Syncope internal storage. So
> the users are created directly in Syncope itself via the REST API.
> Does this kind of deadlock issue ring any bells for any issue fixed
> subsequent to 1.2.5? If not we will try to reproduce the issue
> locally to try to narrow the problem down.
>
> Thanks,
>
> Colm.
>
> On Thu, Mar 17, 2016 at 1:27 PM, Giacomo Lamonaco
> irasa.net> wrote:
> > Il giorno mer, 16/03/2016 alle 15.27 +0000, Aniket Band ha scritto:
> > > Hello All,
> > >
> > > One of our customers uses Syncope (1.2.5) and got some
> > performance
> > > issues. They want to upload 100.000 users via REST interface.
> > They
> > > created 3-5 routes in parallel to call the rest interface.
> > > Unfortunately it requires 2 day to insert 100.000 users. As far
> > as I
> > > know it should not take 2 days. The Log claims some deadlocks in
> > the
> > > database.
> > >
> > > Any idea about it? In the mailing list and in JIRA I could not
> > find
> > > any similar issue.
> > >
> > > Database is a MSSQL in a cluster. Thanks in advance for your
> > inputs.
> > >
> > Hi Aniket,
> >
> > Do you have some external resources connected to Syncope?
> > (Connected means that every time you create an user in Syncope,
> > that user is automatically propagated to the external resource).
> >
> > Do you have the possibility to create users just on one MSSQL
> > instance (switching off the cluster)?
> >
> > It's not always a good idea to perform large 'import' using the
> > REST interface. You could try to perform this kind of import at a
> > lower level, maybe creating a dump containing those users.
> >
> > Best Regards,
> >
> > --
> > Giacomo Lamonaco
> >
> > Tirasa - Open Source Excellence
> > http://www.tirasa.net/
> >
> > Syncope committer
> > http://people.apache.org/~giacomolm
> >
> >
> >
>
>
Re: Deadlocks - Performance Issue
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Giacomo,
My understanding is that there are no external resources involved here. The
MSSQL cluster is used for the Syncope internal storage. So the users are
created directly in Syncope itself via the REST API. Does this kind of
deadlock issue ring any bells for any issue fixed subsequent to 1.2.5? If
not we will try to reproduce the issue locally to try to narrow the problem
down.
Thanks,
Colm.
On Thu, Mar 17, 2016 at 1:27 PM, Giacomo Lamonaco <
giacomo.lamonaco@tirasa.net> wrote:
> Il giorno mer, 16/03/2016 alle 15.27 +0000, Aniket Band ha scritto:
> > Hello All,
> >
> > One of our customers uses Syncope (1.2.5) and got some performance
> > issues. They want to upload 100.000 users via REST interface. They
> > created 3-5 routes in parallel to call the rest interface.
> > Unfortunately it requires 2 day to insert 100.000 users. As far as I
> > know it should not take 2 days. The Log claims some deadlocks in the
> > database.
> >
> > Any idea about it? In the mailing list and in JIRA I could not find
> > any similar issue.
> >
> > Database is a MSSQL in a cluster. Thanks in advance for your inputs.
> >
> Hi Aniket,
>
> Do you have some external resources connected to Syncope? (Connected means
> that every time you create an user in Syncope, that user is automatically
> propagated to the external resource).
>
> Do you have the possibility to create users just on one MSSQL instance
> (switching off the cluster)?
>
> It's not always a good idea to perform large 'import' using the REST
> interface. You could try to perform this kind of import at a lower level,
> maybe creating a dump containing those users.
>
> Best Regards,
>
> --
> Giacomo Lamonaco
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Syncope committer
> http://people.apache.org/~giacomolm
>
>
>
--
Colm O hEigeartaigh
Talend Community Coder
http://coders.talend.com
Re: Deadlocks - Performance Issue
Posted by Giacomo Lamonaco <gi...@tirasa.net>.
Il giorno mer, 16/03/2016 alle 15.27 +0000, Aniket Band ha scritto:
> Hello All,
>
> One of our customers uses Syncope (1.2.5) and got some performance
> issues. They want to upload 100.000 users via REST interface. They
> created 3-5 routes in parallel to call the rest interface.
> Unfortunately it requires 2 day to insert 100.000 users. As far as I
> know it should not take 2 days. The Log claims some deadlocks in the
> database.
>
> Any idea about it? In the mailing list and in JIRA I could not find
> any similar issue.
>
> Database is a MSSQL in a cluster. Thanks in advance for your inputs.
>
Hi Aniket,
Do you have some external resources connected to Syncope? (Connected means that every time you create an user in Syncope, that user is automatically propagated to the external resource).
Do you have the possibility to create users just on one MSSQL instance (switching off the cluster)?
It's not always a good idea to perform large 'import' using the REST interface. You could try to perform this kind of import at a lower level, maybe creating a dump containing those users.
Best Regards,
--
Giacomo Lamonaco
Tirasa - Open Source Excellence
http://www.tirasa.net/
Syncope committer
http://people.apache.org/~giacomolm