You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@syncope.apache.org by utpal kas <ut...@yahoo.com> on 2021/01/07 18:04:48 UTC
Syncope and Hikari
Hello,
We are facing an unique situation,
The Master.properties file kept under (/apache_home/webapps/syncope/WEB-INF/classes/domains) folder has following connection properties defined.
Master.pool.maxActive=20Master.pool.minIdle=4
We enabled the debug log for hikari, and during the provision process we observed the syncope/hikari is using only 8 active connections, not sure from where this number is coming (MasterDomain.xml's default maxConnection value updated to 20 as well). We are getting timeout failures when the load is around 20K users or so and the processing sometimes totally stops midway after throwing connection not available exception or sometimes continue (may be due to remediation check box is true in pull task), dont know why. We tried both Full and Incremental mode.
Anything wrongly set interms of connection setting? We are using 2.1.5 v. Your guidance is appreciated.
Thanks,Utpal
Re: Syncope: Consolidated propagation report
Posted by utpal kas <ut...@yahoo.com>.
That helps, Thank you Francesco!
On Wednesday, March 10, 2021, 02:50:30 AM EST, Francesco Chicchiriccò <il...@apache.org> wrote:
On 08/03/21 20:11, utpal kas wrote:
Hello,
We have set up the PULL task to execute auto-propagation to LDAP (external resource).
After the PULL task's execution we are getting the Summary result by invoking syncope/rest/tasks/executions/recent endpoint. The summary shows, how many users are create/updated/deleted. I do see the syncope/rest/tasks/executions/recent endpoint does show the PROPAGATION task related records but they are based on individual user.
We like to have a summary report ( how many users are create/updated/deleted) generated for PROPAGATION task, is there any easy way?
Hi Utpal,
what you are asking for is not available OOTB: compared to Pull (which is related to a set of objects being received from an External Resource), Propagation is instead related to a single identity being sent from Syncope to External Resources.
You might however extract the information you are looking for from the following REST endpoint:
/tasks/PROPAGATION?page=1&size=25&orderby=end%20DESC&resource=the-resource&anyTypeKind=USER&details=true
this will return the first matching 25 propagation tasks for users on the "the-resource" External Resource, with executions (details), sorted by end date (completion).
HTH
Regards.
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: Syncope: Consolidated propagation report
Posted by utpal kas <ut...@yahoo.com>.
Thanks Francesco, for clarification.
On Friday, March 12, 2021, 07:41:56 AM EST, Francesco Chicchiriccò <il...@apache.org> wrote:
On 10/03/21 13:30, utpal kas wrote:
Another question,
Is there any way to get the details of all the propagation happened today?
Something like, if yes, can you please tell me which format I should send the date?
/tasks/PROPAGATION?end=<today's date>
As you can find out by visiting the Tasks endpoint in Swagger UI, it is not possible to filter tasks by start / end date.
Regards.
On Wednesday, March 10, 2021, 02:50:30 AM EST, Francesco Chicchiriccò <il...@apache.org> wrote:
On 08/03/21 20:11, utpal kas wrote:
Hello,
We have set up the PULL task to execute auto-propagation to LDAP (external resource).
After the PULL task's execution we are getting the Summary result by invoking syncope/rest/tasks/executions/recent endpoint. The summary shows, how many users are create/updated/deleted. I do see the syncope/rest/tasks/executions/recent endpoint does show the PROPAGATION task related records but they are based on individual user.
We like to have a summary report ( how many users are create/updated/deleted) generated for PROPAGATION task, is there any easy way?
Hi Utpal,
what you are asking for is not available OOTB: compared to Pull (which is related to a set of objects being received from an External Resource), Propagation is instead related to a single identity being sent from Syncope to External Resources.
You might however extract the information you are looking for from the following REST endpoint:
/tasks/PROPAGATION?page=1&size=25&orderby=end%20DESC&resource=the-resource&anyTypeKind=USER&details=true
this will return the first matching 25 propagation tasks for users on the "the-resource" External Resource, with executions (details), sorted by end date (completion).
HTH
Regards.
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: Syncope: Consolidated propagation report
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 10/03/21 13:30, utpal kas wrote:
> Another question,
>
> Is there any way to get the details of all the propagation happened today?
>
> Something like, if yes, can you please tell me which format I should send the date?
>
> /tasks/PROPAGATION?end=<today's date>
As you can find out by visiting the Tasks endpoint in Swagger UI, it is not possible to filter tasks by start / end date.
Regards.
> On Wednesday, March 10, 2021, 02:50:30 AM EST, Francesco Chicchiriccò <il...@apache.org> wrote:
>
>
> On 08/03/21 20:11, utpal kas wrote:
> Hello,
>
> We have set up the PULL task to execute auto-propagation to LDAP (external resource).
>
> After the PULL task's execution we are getting the Summary result by invoking *syncope/rest/tasks/executions/recent *endpoint. The summary shows, how many users are create/updated/deleted. I do see the *syncope/rest/tasks/executions/recent *endpoint**does show the *PROPAGATION *task related records but they are based on individual user.
>
> We like to have a summary report ( how many users are create/updated/deleted) generated for PROPAGATION task, is there any easy way?
>
> Hi Utpal,
> what you are asking for is not available OOTB: compared to Pull (which is related to a set of objects being received from an External Resource), Propagation is instead related to a single identity being sent from Syncope to External Resources.
>
> You might however extract the information you are looking for from the following REST endpoint:
>
> /tasks/PROPAGATION?page=1&size=25&orderby=end%20DESC&resource=the-resource&anyTypeKind=USER&details=true
>
> this will return the first matching 25 propagation tasks for users on the "the-resource" External Resource, with executions (details), sorted by end date (completion).
>
> HTH
> Regards.
>
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: Syncope: Consolidated propagation report
Posted by utpal kas <ut...@yahoo.com>.
Another question,
Is there any way to get the details of all the propagation happened today?
Something like, if yes, can you please tell me which format I should send the date?
/tasks/PROPAGATION?end=<today's date>
On Wednesday, March 10, 2021, 02:50:30 AM EST, Francesco Chicchiriccò <il...@apache.org> wrote:
On 08/03/21 20:11, utpal kas wrote:
Hello,
We have set up the PULL task to execute auto-propagation to LDAP (external resource).
After the PULL task's execution we are getting the Summary result by invoking syncope/rest/tasks/executions/recent endpoint. The summary shows, how many users are create/updated/deleted. I do see the syncope/rest/tasks/executions/recent endpoint does show the PROPAGATION task related records but they are based on individual user.
We like to have a summary report ( how many users are create/updated/deleted) generated for PROPAGATION task, is there any easy way?
Hi Utpal,
what you are asking for is not available OOTB: compared to Pull (which is related to a set of objects being received from an External Resource), Propagation is instead related to a single identity being sent from Syncope to External Resources.
You might however extract the information you are looking for from the following REST endpoint:
/tasks/PROPAGATION?page=1&size=25&orderby=end%20DESC&resource=the-resource&anyTypeKind=USER&details=true
this will return the first matching 25 propagation tasks for users on the "the-resource" External Resource, with executions (details), sorted by end date (completion).
HTH
Regards.
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: Syncope: Consolidated propagation report
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 08/03/21 20:11, utpal kas wrote:
> Hello,
>
> We have set up the PULL task to execute auto-propagation to LDAP (external resource).
>
> After the PULL task's execution we are getting the Summary result by invoking *syncope/rest/tasks/executions/recent *endpoint. The summary shows, how many users are create/updated/deleted. I do see the *syncope/rest/tasks/executions/recent *endpoint**does show the *PROPAGATION *task related records but they are based on individual user.
>
> We like to have a summary report ( how many users are create/updated/deleted) generated for PROPAGATION task, is there any easy way?
Hi Utpal,
what you are asking for is not available OOTB: compared to Pull (which is related to a set of objects being received from an External Resource), Propagation is instead related to a single identity being sent from Syncope to External Resources.
You might however extract the information you are looking for from the following REST endpoint:
/tasks/PROPAGATION?page=1&size=25&orderby=end%20DESC&resource=the-resource&anyTypeKind=USER&details=true
this will return the first matching 25 propagation tasks for users on the "the-resource" External Resource, with executions (details), sorted by end date (completion).
HTH
Regards.
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Syncope: Consolidated propagation report
Posted by utpal kas <ut...@yahoo.com>.
Hello,
We have set up the PULL task to execute auto-propagation to LDAP (external resource).
After the PULL task's execution we are getting the Summary result by invoking syncope/rest/tasks/executions/recent endpoint. The summary shows, how many users are create/updated/deleted. I do see the syncope/rest/tasks/executions/recent endpoint does show the PROPAGATION task related records but they are based on individual user.
We like to have a summary report ( how many users are create/updated/deleted) generated for PROPAGATION task, is there any easy way?
Thanks,
Utpal
Re: Syncope and Hikari
Posted by utpal kas <ut...@yahoo.com>.
Thank you Francesco for the explanation.
Utpal.
On Monday, January 11, 2021, 03:49:09 AM EST, Francesco Chicchiriccò <il...@apache.org> wrote:
On 07/01/21 19:04, utpal kas wrote:
Hello,
We are facing an unique situation,
The Master.properties file kept under (/apache_home/webapps/syncope/WEB-INF/classes/domains) folder has following connection properties defined.
Master.pool.maxActive=20 Master.pool.minIdle=4
We enabled the debug log for hikari, and during the provision process we observed the syncope/hikari is using only 8 active connections, not sure from where this number is coming (MasterDomain.xml's default maxConnection value updated to 20 as well). We are getting timeout failures when the load is around 20K users or so and the processing sometimes totally stops midway after throwing connection not available exception or sometimes continue (may be due to remediation check box is true in pull task), dont know why. We tried both Full and Incremental mode.
Anything wrongly set interms of connection setting? We are using 2.1.5 v. Your guidance is appreciated.
Hi Utpal,
nothing unexpected is happening here.
When you did build your Maven Syncope-based project to generate WAR artifacts for deploy, you should have passed some arguments to mvn, then copied the relevant configuration files to the chose deployment directories, as indicated by [1], e.g.
$ mvn clean verify \
-Dconf.directory=/opt/syncope/conf \
-Dbundles.directory=/opt/syncope/bundles \
-Dlog.directory=/opt/syncope/log
$ cp core/target/classes/*properties /opt/syncope/conf
$ cp console/target/classes/*properties /opt/syncope/conf
$ cp enduser/target/classes/*properties /opt/syncope/conf
$ cp enduser/target/classes/customFormAttributes.json /opt/syncope/conf
$ cp enduser/target/classes/customTemplate.json /opt/syncope/conf
This means that the relevant Master.properties file (where the Hikari pool parameters are set) is *not* the one inside the WAR but rather the one under the configured conf.directory - /opt/syncope/conf in the sample above.
Same applies to all other files.
Hope this clarifies.
Regards.
[1]http://syncope.apache.org/docs/2.1/reference-guide.html#deployment-directories
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: Syncope and Hikari
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 07/01/21 19:04, utpal kas wrote:
> Hello,
>
> We are facing an unique situation,
>
> The Master.properties file kept under (/apache_home/webapps/syncope/WEB-INF/classes/domains) folder has following connection properties defined.
>
> Master.pool.maxActive=20
> Master.pool.minIdle=4
>
> We enabled the debug log for hikari, and during the provision process we observed the syncope/hikari is using only 8 active connections, not sure from where this number is coming (MasterDomain.xml's default maxConnection value updated to 20 as well).
> We are getting timeout failures when the load is around 20K users or so and the processing sometimes totally stops midway after throwing connection not available exception or sometimes continue (may be due to remediation check box is true in pull task), dont know why. We tried both Full and Incremental mode.
>
> Anything wrongly set interms of connection setting? We are using 2.1.5 v. Your guidance is appreciated.
Hi Utpal,
nothing unexpected is happening here.
When you did build your Maven Syncope-based project to generate WAR artifacts for deploy, you should have passed some arguments to mvn, then copied the relevant configuration files to the chose deployment directories, as indicated by [1], e.g.
$ mvn clean verify \
-Dconf.directory=/opt/syncope/conf \
-Dbundles.directory=/opt/syncope/bundles \
-Dlog.directory=/opt/syncope/log
$ cp core/target/classes/*properties /opt/syncope/conf
$ cp console/target/classes/*properties /opt/syncope/conf
$ cp enduser/target/classes/*properties /opt/syncope/conf
$ cp enduser/target/classes/customFormAttributes.json /opt/syncope/conf
$ cp enduser/target/classes/customTemplate.json /opt/syncope/conf
This means that the relevant Master.properties file (where the Hikari pool parameters are set) is *not* the one inside the WAR but rather the one under the configured conf.directory - /opt/syncope/conf in the sample above.
Same applies to all other files.
Hope this clarifies.
Regards.
[1] http://syncope.apache.org/docs/2.1/reference-guide.html#deployment-directories
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/