You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@syncope.apache.org by Rob van de Meulengraaf <ro...@finalist.com> on 2015/03/06 13:35:54 UTC

Problem with scriptedsql connector with quick update after create

Hi,

Our client is having some problems with the ScriptedSQL connector. In some cases it tries to do a create in an external database instead of the expected update. This will throw an exception that the entry is already existing.

Our flow is:
1. Create account 1
2. Create account 2
3. Update attribute in account 1 with the ID from account 2 (link account2 to account1)

In can see that in AbstractPropagationTaskExecutor the flow is:
1. find entry to be updated
2. if not found do create
3. if found do update

Apparently there are some timing issues between the find and the create/update when doing two propagations quick after each other. They probably run in parallel? I does not happen always, only under some load.

I subclassed PriorityPropagationTaskExecutor and have overridden the execute method with a synchronized version and this seems to stop the problem:

	@Override
	public synchronized TaskExec execute(PropagationTask task, PropagationReporter reporter) {
		return super.execute(task, reporter);
	}

However that will probably slow down as all propagations are now sequential?

Any ideas for a better solution or workaround? Or is my solution correct?

We use Syncope 1.1.8

Thanks,
Rob

Re: Problem with scriptedsql connector with quick update after create

Posted by Francesco Chicchiriccò <il...@apache.org>.

On 06/03/2015 13:35, Rob van de Meulengraaf wrote:
> Hi,
>
> Our client is having some problems with the ScriptedSQL connector. In 
> some cases it tries to do a create in an external database instead of 
> the expected update. This will throw an exception that the entry is 
> already existing.
>
> Our flow is:
> 1. Create account 1
> 2. Create account 2
> 3. Update attribute in account 1 with the ID from account 2 (link 
> account2 to account1)
>
> In can see that in AbstractPropagationTaskExecutor the flow is:
> 1. find entry to be updated
> 2. if not found do create
> 3. if found do update
>
> Apparently there are some timing issues between the find and the 
> create/update when doing two propagations quick after each other. They 
> probably run in parallel? I does not happen always, only under some load.
>
> I subclassed PriorityPropagationTaskExecutor and have overridden the 
> execute method with a synchronized version and this seems to stop the 
> problem:
>
> @Override
> public synchronized TaskExec execute(PropagationTask task, 
> PropagationReporter reporter) {
> return super.execute(task, reporter);
> }
>
> However that will probably slow down as all propagations are now 
> sequential?
>
> Any ideas for a better solution or workaround? Or is my solution correct?
>
> We use Syncope 1.1.8

Hi Rob,
in principle, the whole propagation process is synchronous, as the 
PropagationTaskExecutor instance that is in use by a given deployment 
(PriorityPropagationTaskExecutor by default, or a subclass as in your 
case) is generally looping over all defined PropagationTask instances to 
be executed and invoking exactly the method you override above.

However, a PropagationTask execution will end up in calling the 
underlying connector, and such invocation is instead delegated to a 
Future task returned by AsyncConnectorFacade, which relies on Spring's 
@Async (this mainly for the ability to define a timeout on effective 
connector invocation).

So, briefly: I don't believe your solution is wrong, even if I am not 
100% convinced it is the best solution for the general case. As 
explained, anyway, I don't see problems for slowing down the propagation 
process as you wonder above.

Regards.

-- 
Francesco Chicchiriccò

Tirasa - Open Source Excellence
http://www.tirasa.net/

Involved at The Apache Software Foundation:
member, Syncope PMC chair, Cocoon PMC, Olingo PMC
http://people.apache.org/~ilgrosso/

Re: Problem with scriptedsql connector with quick update after create

Posted by Rob van de Meulengraaf <ro...@finalist.com>.

Ciao Fabio,

Our client is using a little bit weird construction as that every main account can have multiple linked subaccounts (I didn’t invent it;). For this reason when a subaccount is created the first time a linked main account (with different attributes) is also created. The two are linked by setting the id of the other in an attribute, like this:

main-account
id: 1
primary-sub-account-id: 2

sub-account
id: 2
main-account-id: 1

Because we do not have the id’s before an account is created the flow is:

1. create main-account
2. get generated id from main-account
3. create sub-account with attribute main-account-id = id from step 2
4. get generated id from sub-account
5. update main-account by setting attribute primary-sub-account-id = id from step 4

So step 1 and 5 will trigger two provisioning actions on the same record in a very short time.

Probably not what Syncope was designed for….

saluti,
Rob

On 6 mrt. 2015, at 14:28, Fabio Martelli <fa...@gmail.com> wrote:

> Il 06/03/2015 13:35, Rob van de Meulengraaf ha scritto:
>> Hi,
>> 
>> Our client is having some problems with the ScriptedSQL connector. In some cases it tries to do a create in an external database instead of the expected update. This will throw an exception that the entry is already existing.
>> 
>> Our flow is:
>> 1. Create account 1
>> 2. Create account 2
>> 3. Update attribute in account 1 with the ID from account 2 (link account2 to account1)
>> 
>> In can see that in AbstractPropagationTaskExecutor the flow is:
>> 1. find entry to be updated
>> 2. if not found do create
>> 3. if found do update
>> 
>> Apparently there are some timing issues between the find and the create/update when doing two propagations quick after each other. They probably run in parallel? I does not happen always, only under some load.
>> 
>> I subclassed PriorityPropagationTaskExecutor and have overridden the execute method with a synchronized version and this seems to stop the problem:
>> 
>>  @Override
>>  public synchronized TaskExec execute(PropagationTask task, PropagationReporter reporter) {
>>  return super.execute(task, reporter);
>>  }
>> 
>> However that will probably slow down as all propagations are now sequential?
>> 
>> Any ideas for a better solution or workaround? Or is my solution correct?
>> 
>> We use Syncope 1.1.8
>> 
>> Thanks,
>> Rob
> Hi Rob, this sounds a little bit weird. Before apply your solution I do think you have to try to find out the real cause of your problem.
> Can you explain better why you have two propagation tasks executed concurrently on the same resource/account?
> Can you provide more details about your configuration?
> 
> Best regards,
> F.
> 
> -- 
> Fabio Martelli
> 
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
> 
> Apache Syncope PMC
> http://people.apache.org/~fmartelli/

Re: Problem with scriptedsql connector with quick update after create

Posted by Fabio Martelli <fa...@gmail.com>.

Il 06/03/2015 13:35, Rob van de Meulengraaf ha scritto:
> Hi,
>
> Our client is having some problems with the ScriptedSQL connector. In 
> some cases it tries to do a create in an external database instead of 
> the expected update. This will throw an exception that the entry is 
> already existing.
>
> Our flow is:
> 1. Create account 1
> 2. Create account 2
> 3. Update attribute in account 1 with the ID from account 2 (link 
> account2 to account1)
>
> In can see that in AbstractPropagationTaskExecutor the flow is:
> 1. find entry to be updated
> 2. if not found do create
> 3. if found do update
>
> Apparently there are some timing issues between the find and the 
> create/update when doing two propagations quick after each other. They 
> probably run in parallel? I does not happen always, only under some load.
>
> I subclassed PriorityPropagationTaskExecutor and have overridden the 
> execute method with a synchronized version and this seems to stop the 
> problem:
>
> @Override
> public synchronized TaskExec execute(PropagationTask task, 
> PropagationReporter reporter) {
> return super.execute(task, reporter);
> }
>
> However that will probably slow down as all propagations are now 
> sequential?
>
> Any ideas for a better solution or workaround? Or is my solution correct?
>
> We use Syncope 1.1.8
>
> Thanks,
> Rob
Hi Rob, this sounds a little bit weird. Before apply your solution I do 
think you have to try to find out the real cause of your problem.
Can you explain better why you have two propagation tasks executed 
concurrently on the same resource/account?
Can you provide more details about your configuration?

Best regards,
F.

-- 
Fabio Martelli

Tirasa - Open Source Excellence
http://www.tirasa.net/

Apache Syncope PMC
http://people.apache.org/~fmartelli/