[DISCUSS] Propagation task executions and no-SQL persistence

[Andrea Patricelli <an...@apache.org>]

Good morning all,

In my experience, in almost each Syncope project I worked on, I had to 
face sooner or later with the huge size of the tables storing 
propagation tasks and their executions (TASK and TASKEXEC). Despite the 
possibility to tune execution storage and verbosity, in the first phases 
of the projects (in test or even in production mode) it is useful to set 
the most verborse trace level for CRUD operations and sometimes also for 
pull/push ones. This because information stored (mainly) into the 
propagation tasks executions are very useful for debugging purposes and 
to understand what happened in case of errors, but also in case of 
successful operations.

Thus most of the times, due to the too large size of TASK and TASKEXEC 
tables, we are forced to clean up them (usually by date) to speed up 
queries on propagation tasks and to make the console section that shows 
resource propagation tasks usable.

Moreover in some cases it is very useful to analyze execution details, 
that for propagation tasks, for example, contain the payload (in JSON 
format) sent to the resource. At the moment there isn't the possibility 
to parse and to make granular queries on task executions.

This said, I would suggest to add to Syncope the possibility to manage 
such information, at least the ones into task executions (TASKEXEC 
table) as big data or, generally speaking, as no SQL information. The 
nearest solution could be the use of Elasticsearch as a possible 
storage, mainly because it has successfuly been embedded into Syncope 
[1] and is easily queriable (also through a free console as Kibana [2]).
Obviously no-SQL world is wide, so we can consider usign another 
technology as MongoDB, a graph database, ecc. But, sincerely speaking, I 
do not see the need to add another technology and dependency(ies) to the 
project.

Lastly, I'm not suggesting to implement it right now, but at least to 
discuss about to add this feature to the roadmap or not.

WDYT?

Best regards,
Andrea

[1] https://syncope.apache.org/docs/2.1/reference-guide.html#elasticsearch

[2] https://www.elastic.co/kibana

-- 
Dott. Andrea Patricelli
Tel. +39 3204524292

Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net

Apache Syncope PMC Member