You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@syncope.apache.org by lfinch <An...@CereCore.net> on 2019/07/29 21:58:37 UTC
Timing differences in pull and push tasks
Hello - I'm running a 2.1.2 implementation and have 9 plain schema defined.
I'm using CSVDIR as connector. Pull task is set to full recon,
matching=update, unmatching = provision (create/update/sync). Push task is
also matching=update, unmatching = provision (create/update/sync). I'm using
the same connector and external resource for push and pull. No fancy
mapping. All very vanilla.
Here are my questions. The pull processes about 160 records per minute,
about 25-26 at a time. The push processes about 32 records per minute, 5 at
a time.
1) We need to improve speed overall. Our initial production load will be
200K. Any suggestions on where we can tweak for higher throughput?
2) Why is there such a different in pull and push task performance?
Thanks!
Lynn
--
Sent from: http://syncope-user.1051894.n5.nabble.com/
Re: Timing differences in pull and push tasks
Posted by lfinch <An...@CereCore.net>.
Hello, Lorenzo! We are using a data transfer system to move information
between various components of a multi-module system. It transfers data
natively in XML, but can accept/output most formats and translate them. We
will not be reading/writing from a database, but will always be picking
up/dropping files in a directory that the data transfer system shares.
Most likely, we will only do a few large loads, so I guess we can live with
the slower throughput. Thanks for your insight and assistance!
--
Sent from: http://syncope-user.1051894.n5.nabble.com/
Re: Timing differences in pull and push tasks
Posted by Lorenzo Di Cola <lo...@tirasa.net>.
Hi Lynn,
Il 31/07/19 17:05, lfinch ha scritto:
> Hi, Lorenzo! Some follow up questions.
>
>> and in general for increase performance I can
> suggest you to not use the CSVDIR as a connector. This connector is not
> the best one in order to have a performing environment
>
> What do you recommend? (It would be nice if we could produce output in XML)
If you want to produce an XML as an output for sure you have to
translate the output of the connector in XML because there is none
connector that produce output in XML. If you are using CSVDIR connector
I think that now you are translating the output from CSV to XML, am I
right?
>
>> There is a performance gap between the pull and the push simply because
> the pull is more complex than the push
>
> That makes sense, but our pull is running at approximately 160 records per
> minute and the push is running about 30 records per minute. I would've
> expected the push to be quicker as well.
In this case I think that the cause of your delay is the CSVDIR
connector: the push is running 30 records per minute (less than the
pull) because you are/writing to file with this connector. For this
reason too I suggest you to not use the CSVDIR connector. An example of
connector that I think can help you is the Scripted SQL one. If you are
using CSV maybe you can translate all your data in a database instance
(some utilities can help you) and so have a connector that do not
read/write from a file but from a database./
// <https://context.reverso.net/traduzione/inglese-italiano/Writing+to+file>
>
>> In order to improve speed overall, are you already using PostgreSQL as a
> DBMS?
>
> We are using MySQL, it's a company standard.
>
> Thank you!
> Lynn
>
>
> --
> Sent from: http://syncope-user.1051894.n5.nabble.com/
Best regards,
--
Dott. Lorenzo Di Cola
Software Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna, 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net
Apache Syncope Committer
http://people.apache.org/phonebook.html?uid=loredicola
Re: Timing differences in pull and push tasks
Posted by lfinch <An...@CereCore.net>.
Hi, Lorenzo! Some follow up questions.
>and in general for increase performance I can
suggest you to not use the CSVDIR as a connector. This connector is not
the best one in order to have a performing environment
What do you recommend? (It would be nice if we could produce output in XML)
>There is a performance gap between the pull and the push simply because
the pull is more complex than the push
That makes sense, but our pull is running at approximately 160 records per
minute and the push is running about 30 records per minute. I would've
expected the push to be quicker as well.
>In order to improve speed overall, are you already using PostgreSQL as a
DBMS?
We are using MySQL, it's a company standard.
Thank you!
Lynn
--
Sent from: http://syncope-user.1051894.n5.nabble.com/
Re: Timing differences in pull and push tasks
Posted by Lorenzo Di Cola <lo...@tirasa.net>.
Hi Lynn,
please read in line.
Il 29/07/19 23:58, lfinch ha scritto:
> Hello - I'm running a 2.1.2 implementation and have 9 plain schema defined.
> I'm using CSVDIR as connector. Pull task is set to full recon,
> matching=update, unmatching = provision (create/update/sync). Push task is
> also matching=update, unmatching = provision (create/update/sync). I'm using
> the same connector and external resource for push and pull. No fancy
> mapping. All very vanilla.
>
> Here are my questions. The pull processes about 160 records per minute,
> about 25-26 at a time. The push processes about 32 records per minute, 5 at
> a time.
>
> 1) We need to improve speed overall. Our initial production load will be
> 200K. Any suggestions on where we can tweak for higher throughput?
For production like yours and in general for increase performance I can
suggest you to not use the CSVDIR as a connector. This connector is not
the best one in order to have a performing environment.
>
> 2) Why is there such a different in pull and push task performance?
There is a performance gap between the pull and the push simply because
the pull is more complex than the push: during the pull there are
matches to do with the Syncope internal entities.
In order to improve speed overall, are you already using PostgreSQL as a
DBMS? Using PostreSQL + JSON can, for sure, increase the performance
like you can see here [1].
You can find a reference in [2] about PostgreSQL and JSONB.
>
> Thanks!
> Lynn
>
> --
> Sent from: http://syncope-user.1051894.n5.nabble.com/
[1] https://www.tirasa.net/blog/benchmarking-apache-syncope-on-postgresql
[2]
https://syncope.apache.org/docs/2.1/reference-guide.html#postgresql-jsonb
Best regards,
--
Dott. Lorenzo Di Cola
Software Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna, 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net
Apache Syncope Committer
http://people.apache.org/phonebook.html?uid=loredicola