You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@syncope.apache.org by Andrea Patricelli <an...@apache.org> on 2019/09/17 07:32:25 UTC
[DISCUSS] Object versioning
Hi all,
in several of the last Syncope projects we recently worked on, we faced
a recurring issue: data inconsistency (at business level) and the need
to know who did what on some object.
For "data inconsistency" I do not mean any issue related to database
records or provisioning, but the values of the user attributes at the
business level (identity info, groups or resources assigned, etc.).
Often a sysadmin wonders about "why this attribute has changed?" or "who
did this (wrong) change, how can I recover the previous value?"
Changes I'm referring to can come from automatic tasks (pull, scheduled)
or manual tasks performed by admin or manager users.
Syncope already provides info about last change date, change password
date or last modifier user, but in my opinion are not enough to
understand the history of the record.
This is why I'm proposing to add to Syncope the possibility of
versioning objects. Something similar has been done for connectors and
resources configurations [1]. It is not a simple work I know :)
Here are some requirements that I was thinking of:
* Versioning should be done on all objects, USER, GROUP and ANY.
* Versioning could be heavy for systems with thousands or million of
users, so it should be a feature to enable/disable at each moment of
the project lifecycle. It should also be tunable, e.g. store only
attributes and not resources, groups, etc.
* In a first release I would leave apart all issues related to
provisioning. I would simply version data stored on Syncope database
* User should have the possibility to see a diff of what has changed
like per connectors and resources configurations.
* User should have the possibility to restore a certain value (or the
whole object?)
* User should be able to see, at least (in a first release), an
history of what has changed on data (e.g. name: foo -> bar) and who
did the change and when.
* User should be able to query by changed data like users who are
subject to "name" change in a certain period (not so useful, maybe).
There are some usefu l open source libraries that support objects
versioning like JaVers [2] or [3], but I'm not sure if this last one
fits all the needs above. If you have any other proposal please don't
hesitate to attach it in this thread.
WDYT?
Have a nice day,
Andrea
[1] https://issues.apache.org/jira/browse/SYNCOPE-1145
[2] https://javers.org/
[3]
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.auditing
--
Dott. Andrea Patricelli
Tel. +39 3204524292
Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net
Apache Syncope PMC Member
Re: [DISCUSS] Object versioning
Posted by Andrea Patricelli <an...@apache.org>.
Il 17/09/19 14:05, Francesco Chicchiriccò ha scritto:
> Hi,
> object versioning looks interesting - even if that might require to use Spring Data JPA - but sounds like quite some work, to me.
>
> One aspect that leaves me not completely convinced is consistency of Users' / Groups' / Any Objects' attributes (from internal storage, e.g. the one subjected to versioning) with data spread across all connected External Resources.
>
> Example:
>
> 1. user 'ilgrosso' with attribute 'salary=1000', mapped to same attribute on LDAP
> 2. an administrator raises that value to 'salary=2000' on Syncope, propagation sends such update to LDAP
> 3. another administrator sees the raised value, thinks 'it's a mistake!' and reverts it to 'salary=1000' by using the object versioning
> 4. LDAP has still 2000
>
> How would you handle such cases?
> Regards.
This is a good point, to add the critical issues of this development.
I think that providing at least the possibility to see (changelog) and
query what happened would be itself a great success :)
About restoring I think that we should consider each restore like an
effective update on the object, so at point 3 restore of 'salary=1000'
should also fire a propagation to LDAP and all associated resources,
otherwise I do not see other solutions to avoid such misalignments.
Best regards,
Andrea
>
> On 17/09/19 09:50, Andrea Patricelli wrote:
>> Hi Misagh,
>>
>> thanks for you interest in the proposal :)
>>
>> Il 17/09/19 09:41, Misagh Moayyed ha scritto:
>>> I think it's a very good idea, and should fit quite nicely with the auditing capabilities of the system to make auditors happy :) I have done some brief work with JaVers myself while I was evaluating it for very similar requirements; it's a an easy-to-use library and also has decent and flexible persistent mechanisms for record keeping; also somewhat Boot friendly. Maybe the two can work with each other in fact; you'd have JaVers as the high-level abstraction that keeps track of history and audits, while the persistence mechanisms are pushed out to different systems, one of which could be handled by Spring Data JPA, while also allowing for MongoDb and etc.
>> Yes I had the chance to work with it several times, too and can confirm the high usability and easy configurability through very minimal code changes.
>>> A few things about history and audits in general that I think would be useful:
>>>
>>> 1. This should be an optional thing; one should have the ability to turn it off and I'd argue that it perhaps should be kept off by default until at least a release passes to avoid surprises.
>>> 2. Audit data can grow quite large depending on system activity and may blow out the storage system or cause performance issues, etc. There should be some sort of "automatic clean up" process that purges records after a reasonable time period, or sufficient warnings that adopters are to do the cleanup themselves.
>> I agree about this two points, data growth is my biggest concern. Very nice idea the cleanup process! I would add them to the requirements list.
>>
>>> --Misagh
>> Best regards,
>> Andrea
>>
>>> ----- Original Message -----
>>>> From: "Andrea Patricelli" <an...@apache.org>
>>>> To: "dev" <de...@syncope.apache.org>
>>>> Sent: Tuesday, September 17, 2019 12:02:25 PM
>>>> Subject: [DISCUSS] Object versioning
>>>> Hi all,
>>>>
>>>> in several of the last Syncope projects we recently worked on, we faced
>>>> a recurring issue: data inconsistency (at business level) and the need
>>>> to know who did what on some object.
>>>>
>>>> For "data inconsistency" I do not mean any issue related to database
>>>> records or provisioning, but the values of the user attributes at the
>>>> business level (identity info, groups or resources assigned, etc.).
>>>> Often a sysadmin wonders about "why this attribute has changed?" or "who
>>>> did this (wrong) change, how can I recover the previous value?"
>>>> Changes I'm referring to can come from automatic tasks (pull, scheduled)
>>>> or manual tasks performed by admin or manager users.
>>>> Syncope already provides info about last change date, change password
>>>> date or last modifier user, but in my opinion are not enough to
>>>> understand the history of the record.
>>>>
>>>> This is why I'm proposing to add to Syncope the possibility of
>>>> versioning objects. Something similar has been done for connectors and
>>>> resources configurations [1]. It is not a simple work I know :)
>>>>
>>>> Here are some requirements that I was thinking of:
>>>>
>>>> * Versioning should be done on all objects, USER, GROUP and ANY.
>>>> * Versioning could be heavy for systems with thousands or million of
>>>> users, so it should be a feature to enable/disable at each moment of
>>>> the project lifecycle. It should also be tunable, e.g. store only
>>>> attributes and not resources, groups, etc.
>>>> * In a first release I would leave apart all issues related to
>>>> provisioning. I would simply version data stored on Syncope database
>>>> * User should have the possibility to see a diff of what has changed
>>>> like per connectors and resources configurations.
>>>> * User should have the possibility to restore a certain value (or the
>>>> whole object?)
>>>> * User should be able to see, at least (in a first release), an
>>>> history of what has changed on data (e.g. name: foo -> bar) and who
>>>> did the change and when.
>>>> * User should be able to query by changed data like users who are
>>>> subject to "name" change in a certain period (not so useful, maybe).
>>>>
>>>> There are some usefu l open source libraries that support objects
>>>> versioning like JaVers [2] or [3], but I'm not sure if this last one
>>>> fits all the needs above. If you have any other proposal please don't
>>>> hesitate to attach it in this thread.
>>>>
>>>> WDYT?
>>>>
>>>> Have a nice day,
>>>> Andrea
>>>>
>>>> [1] https://issues.apache.org/jira/browse/SYNCOPE-1145
>>>>
>>>> [2] https://javers.org/
>>>>
>>>> [3] https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.auditing
--
Dott. Andrea Patricelli
Tel. +39 3204524292
Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net
Apache Syncope PMC Member
Re: [DISCUSS] Object versioning
Posted by "Dalrymple, Philip" <pw...@mdtsoft.com>.
I would say that on any change the change should be sent to all attached resources, the
only place that this might be tricky is for the password.
By chance this is an area that I work in every day, my company sells change management for
manufacturing companies and the whole field is full of pitfalls and corner cases but Francesco's
question (at least what should be done, doing it is a different story) is easy, any change should be
sent to all the connected resources, again the only tricky part is passwords, if a Hash is used for
storage then the change can't be rolled back.
Giving it a bit more thought.
Passwords are an area that should NEVER be rolled
back I think. So the “restore from previous state” function should be disabled for passwords.
The way that I do this in our HR system won't work for a Java Object model type system like
Syncope, I use Postgres triggers to create the reversible audit and change tracking for
any changes to the records.
--
This email, and any files transmitted with it, are confidential
and intended solely for the use of the individual or entity to
whom they are addressed. If you have received this email in error,
please advise postmaster@mdtsoft.com <ma...@mdtsoft.com>.
3480 Preston Ridge Road
Suite 450
Alpharetta, GA 30005
Philip W. Dalrymple III <pw...@mdtsoft.com>
MDT Software - Automation Management Company
+1 678 297 1001
Fax +1 678 297 1003
________________________________________
From: Francesco Chicchiriccò <il...@apache.org>
Sent: Tuesday, September 17, 2019 8:05
To: dev@syncope.apache.org
Subject: Re: [DISCUSS] Object versioning
Hi,
object versioning looks interesting - even if that might require to use Spring Data JPA - but sounds like quite some work, to me.
One aspect that leaves me not completely convinced is consistency of Users' / Groups' / Any Objects' attributes (from internal storage, e.g. the one subjected to versioning) with data spread across all connected External Resources.
Example:
1. user 'ilgrosso' with attribute 'salary=1000', mapped to same attribute on LDAP
2. an administrator raises that value to 'salary=2000' on Syncope, propagation sends such update to LDAP
3. another administrator sees the raised value, thinks 'it's a mistake!' and reverts it to 'salary=1000' by using the object versioning
4. LDAP has still 2000
How would you handle such cases?
Regards.
On 17/09/19 09:50, Andrea Patricelli wrote:
> Hi Misagh,
>
> thanks for you interest in the proposal :)
>
> Il 17/09/19 09:41, Misagh Moayyed ha scritto:
>> I think it's a very good idea, and should fit quite nicely with the auditing capabilities of the system to make auditors happy :) I have done some brief work with JaVers myself while I was evaluating it for very similar requirements; it's a an easy-to-use library and also has decent and flexible persistent mechanisms for record keeping; also somewhat Boot friendly. Maybe the two can work with each other in fact; you'd have JaVers as the high-level abstraction that keeps track of history and audits, while the persistence mechanisms are pushed out to different systems, one of which could be handled by Spring Data JPA, while also allowing for MongoDb and etc.
> Yes I had the chance to work with it several times, too and can confirm the high usability and easy configurability through very minimal code changes.
>>
>> A few things about history and audits in general that I think would be useful:
>>
>> 1. This should be an optional thing; one should have the ability to turn it off and I'd argue that it perhaps should be kept off by default until at least a release passes to avoid surprises.
>> 2. Audit data can grow quite large depending on system activity and may blow out the storage system or cause performance issues, etc. There should be some sort of "automatic clean up" process that purges records after a reasonable time period, or sufficient warnings that adopters are to do the cleanup themselves.
>
> I agree about this two points, data growth is my biggest concern. Very nice idea the cleanup process! I would add them to the requirements list.
>
>> --Misagh
>
> Best regards,
> Andrea
>
>>
>> ----- Original Message -----
>>> From: "Andrea Patricelli" <an...@apache.org>
>>> To: "dev" <de...@syncope.apache.org>
>>> Sent: Tuesday, September 17, 2019 12:02:25 PM
>>> Subject: [DISCUSS] Object versioning
>>> Hi all,
>>>
>>> in several of the last Syncope projects we recently worked on, we faced
>>> a recurring issue: data inconsistency (at business level) and the need
>>> to know who did what on some object.
>>>
>>> For "data inconsistency" I do not mean any issue related to database
>>> records or provisioning, but the values of the user attributes at the
>>> business level (identity info, groups or resources assigned, etc.).
>>> Often a sysadmin wonders about "why this attribute has changed?" or "who
>>> did this (wrong) change, how can I recover the previous value?"
>>> Changes I'm referring to can come from automatic tasks (pull, scheduled)
>>> or manual tasks performed by admin or manager users.
>>> Syncope already provides info about last change date, change password
>>> date or last modifier user, but in my opinion are not enough to
>>> understand the history of the record.
>>>
>>> This is why I'm proposing to add to Syncope the possibility of
>>> versioning objects. Something similar has been done for connectors and
>>> resources configurations [1]. It is not a simple work I know :)
>>>
>>> Here are some requirements that I was thinking of:
>>>
>>> * Versioning should be done on all objects, USER, GROUP and ANY.
>>> * Versioning could be heavy for systems with thousands or million of
>>> users, so it should be a feature to enable/disable at each moment of
>>> the project lifecycle. It should also be tunable, e.g. store only
>>> attributes and not resources, groups, etc.
>>> * In a first release I would leave apart all issues related to
>>> provisioning. I would simply version data stored on Syncope database
>>> * User should have the possibility to see a diff of what has changed
>>> like per connectors and resources configurations.
>>> * User should have the possibility to restore a certain value (or the
>>> whole object?)
>>> * User should be able to see, at least (in a first release), an
>>> history of what has changed on data (e.g. name: foo -> bar) and who
>>> did the change and when.
>>> * User should be able to query by changed data like users who are
>>> subject to "name" change in a certain period (not so useful, maybe).
>>>
>>> There are some usefu l open source libraries that support objects
>>> versioning like JaVers [2] or [3], but I'm not sure if this last one
>>> fits all the needs above. If you have any other proposal please don't
>>> hesitate to attach it in this thread.
>>>
>>> WDYT?
>>>
>>> Have a nice day,
>>> Andrea
>>>
>>> [1] https://issues.apache.org/jira/browse/SYNCOPE-1145
>>>
>>> [2] https://javers.org/
>>>
>>> [3] https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.auditing
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: [DISCUSS] Object versioning
Posted by Francesco Chicchiriccò <il...@apache.org>.
Hi,
object versioning looks interesting - even if that might require to use Spring Data JPA - but sounds like quite some work, to me.
One aspect that leaves me not completely convinced is consistency of Users' / Groups' / Any Objects' attributes (from internal storage, e.g. the one subjected to versioning) with data spread across all connected External Resources.
Example:
1. user 'ilgrosso' with attribute 'salary=1000', mapped to same attribute on LDAP
2. an administrator raises that value to 'salary=2000' on Syncope, propagation sends such update to LDAP
3. another administrator sees the raised value, thinks 'it's a mistake!' and reverts it to 'salary=1000' by using the object versioning
4. LDAP has still 2000
How would you handle such cases?
Regards.
On 17/09/19 09:50, Andrea Patricelli wrote:
> Hi Misagh,
>
> thanks for you interest in the proposal :)
>
> Il 17/09/19 09:41, Misagh Moayyed ha scritto:
>> I think it's a very good idea, and should fit quite nicely with the auditing capabilities of the system to make auditors happy :) I have done some brief work with JaVers myself while I was evaluating it for very similar requirements; it's a an easy-to-use library and also has decent and flexible persistent mechanisms for record keeping; also somewhat Boot friendly. Maybe the two can work with each other in fact; you'd have JaVers as the high-level abstraction that keeps track of history and audits, while the persistence mechanisms are pushed out to different systems, one of which could be handled by Spring Data JPA, while also allowing for MongoDb and etc.
> Yes I had the chance to work with it several times, too and can confirm the high usability and easy configurability through very minimal code changes.
>>
>> A few things about history and audits in general that I think would be useful:
>>
>> 1. This should be an optional thing; one should have the ability to turn it off and I'd argue that it perhaps should be kept off by default until at least a release passes to avoid surprises.
>> 2. Audit data can grow quite large depending on system activity and may blow out the storage system or cause performance issues, etc. There should be some sort of "automatic clean up" process that purges records after a reasonable time period, or sufficient warnings that adopters are to do the cleanup themselves.
>
> I agree about this two points, data growth is my biggest concern. Very nice idea the cleanup process! I would add them to the requirements list.
>
>> --Misagh
>
> Best regards,
> Andrea
>
>>
>> ----- Original Message -----
>>> From: "Andrea Patricelli" <an...@apache.org>
>>> To: "dev" <de...@syncope.apache.org>
>>> Sent: Tuesday, September 17, 2019 12:02:25 PM
>>> Subject: [DISCUSS] Object versioning
>>> Hi all,
>>>
>>> in several of the last Syncope projects we recently worked on, we faced
>>> a recurring issue: data inconsistency (at business level) and the need
>>> to know who did what on some object.
>>>
>>> For "data inconsistency" I do not mean any issue related to database
>>> records or provisioning, but the values of the user attributes at the
>>> business level (identity info, groups or resources assigned, etc.).
>>> Often a sysadmin wonders about "why this attribute has changed?" or "who
>>> did this (wrong) change, how can I recover the previous value?"
>>> Changes I'm referring to can come from automatic tasks (pull, scheduled)
>>> or manual tasks performed by admin or manager users.
>>> Syncope already provides info about last change date, change password
>>> date or last modifier user, but in my opinion are not enough to
>>> understand the history of the record.
>>>
>>> This is why I'm proposing to add to Syncope the possibility of
>>> versioning objects. Something similar has been done for connectors and
>>> resources configurations [1]. It is not a simple work I know :)
>>>
>>> Here are some requirements that I was thinking of:
>>>
>>> * Versioning should be done on all objects, USER, GROUP and ANY.
>>> * Versioning could be heavy for systems with thousands or million of
>>> users, so it should be a feature to enable/disable at each moment of
>>> the project lifecycle. It should also be tunable, e.g. store only
>>> attributes and not resources, groups, etc.
>>> * In a first release I would leave apart all issues related to
>>> provisioning. I would simply version data stored on Syncope database
>>> * User should have the possibility to see a diff of what has changed
>>> like per connectors and resources configurations.
>>> * User should have the possibility to restore a certain value (or the
>>> whole object?)
>>> * User should be able to see, at least (in a first release), an
>>> history of what has changed on data (e.g. name: foo -> bar) and who
>>> did the change and when.
>>> * User should be able to query by changed data like users who are
>>> subject to "name" change in a certain period (not so useful, maybe).
>>>
>>> There are some usefu l open source libraries that support objects
>>> versioning like JaVers [2] or [3], but I'm not sure if this last one
>>> fits all the needs above. If you have any other proposal please don't
>>> hesitate to attach it in this thread.
>>>
>>> WDYT?
>>>
>>> Have a nice day,
>>> Andrea
>>>
>>> [1] https://issues.apache.org/jira/browse/SYNCOPE-1145
>>>
>>> [2] https://javers.org/
>>>
>>> [3] https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.auditing
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: [DISCUSS] Object versioning
Posted by Andrea Patricelli <an...@apache.org>.
Hi Misagh,
thanks for you interest in the proposal :)
Il 17/09/19 09:41, Misagh Moayyed ha scritto:
> I think it's a very good idea, and should fit quite nicely with the auditing capabilities of the system to make auditors happy :) I have done some brief work with JaVers myself while I was evaluating it for very similar requirements; it's a an easy-to-use library and also has decent and flexible persistent mechanisms for record keeping; also somewhat Boot friendly. Maybe the two can work with each other in fact; you'd have JaVers as the high-level abstraction that keeps track of history and audits, while the persistence mechanisms are pushed out to different systems, one of which could be handled by Spring Data JPA, while also allowing for MongoDb and etc.
Yes I had the chance to work with it several times, too and can confirm
the high usability and easy configurability through very minimal code
changes.
>
> A few things about history and audits in general that I think would be useful:
>
> 1. This should be an optional thing; one should have the ability to turn it off and I'd argue that it perhaps should be kept off by default until at least a release passes to avoid surprises.
> 2. Audit data can grow quite large depending on system activity and may blow out the storage system or cause performance issues, etc. There should be some sort of "automatic clean up" process that purges records after a reasonable time period, or sufficient warnings that adopters are to do the cleanup themselves.
I agree about this two points, data growth is my biggest concern. Very
nice idea the cleanup process! I would add them to the requirements list.
> --Misagh
Best regards,
Andrea
>
> ----- Original Message -----
>> From: "Andrea Patricelli" <an...@apache.org>
>> To: "dev" <de...@syncope.apache.org>
>> Sent: Tuesday, September 17, 2019 12:02:25 PM
>> Subject: [DISCUSS] Object versioning
>> Hi all,
>>
>> in several of the last Syncope projects we recently worked on, we faced
>> a recurring issue: data inconsistency (at business level) and the need
>> to know who did what on some object.
>>
>> For "data inconsistency" I do not mean any issue related to database
>> records or provisioning, but the values of the user attributes at the
>> business level (identity info, groups or resources assigned, etc.).
>> Often a sysadmin wonders about "why this attribute has changed?" or "who
>> did this (wrong) change, how can I recover the previous value?"
>> Changes I'm referring to can come from automatic tasks (pull, scheduled)
>> or manual tasks performed by admin or manager users.
>> Syncope already provides info about last change date, change password
>> date or last modifier user, but in my opinion are not enough to
>> understand the history of the record.
>>
>> This is why I'm proposing to add to Syncope the possibility of
>> versioning objects. Something similar has been done for connectors and
>> resources configurations [1]. It is not a simple work I know :)
>>
>> Here are some requirements that I was thinking of:
>>
>> * Versioning should be done on all objects, USER, GROUP and ANY.
>> * Versioning could be heavy for systems with thousands or million of
>> users, so it should be a feature to enable/disable at each moment of
>> the project lifecycle. It should also be tunable, e.g. store only
>> attributes and not resources, groups, etc.
>> * In a first release I would leave apart all issues related to
>> provisioning. I would simply version data stored on Syncope database
>> * User should have the possibility to see a diff of what has changed
>> like per connectors and resources configurations.
>> * User should have the possibility to restore a certain value (or the
>> whole object?)
>> * User should be able to see, at least (in a first release), an
>> history of what has changed on data (e.g. name: foo -> bar) and who
>> did the change and when.
>> * User should be able to query by changed data like users who are
>> subject to "name" change in a certain period (not so useful, maybe).
>>
>> There are some usefu l open source libraries that support objects
>> versioning like JaVers [2] or [3], but I'm not sure if this last one
>> fits all the needs above. If you have any other proposal please don't
>> hesitate to attach it in this thread.
>>
>> WDYT?
>>
>> Have a nice day,
>> Andrea
>>
>> [1] https://issues.apache.org/jira/browse/SYNCOPE-1145
>>
>> [2] https://javers.org/
>>
>> [3]
>> https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.auditing
>>
>> --
>> Dott. Andrea Patricelli
>> Tel. +39 3204524292
>>
>> Engineer @ Tirasa S.r.l.
>> Viale Vittoria Colonna 97 - 65127 Pescara
>> Tel +39 0859116307 / FAX +39 0859111173
>> http://www.tirasa.net
>>
>> Apache Syncope PMC Member
--
Dott. Andrea Patricelli
Tel. +39 3204524292
Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net
Apache Syncope PMC Member
Re: [DISCUSS] Object versioning
Posted by Misagh Moayyed <mi...@tirasa.net>.
I think it's a very good idea, and should fit quite nicely with the auditing capabilities of the system to make auditors happy :) I have done some brief work with JaVers myself while I was evaluating it for very similar requirements; it's a an easy-to-use library and also has decent and flexible persistent mechanisms for record keeping; also somewhat Boot friendly. Maybe the two can work with each other in fact; you'd have JaVers as the high-level abstraction that keeps track of history and audits, while the persistence mechanisms are pushed out to different systems, one of which could be handled by Spring Data JPA, while also allowing for MongoDb and etc.
A few things about history and audits in general that I think would be useful:
1. This should be an optional thing; one should have the ability to turn it off and I'd argue that it perhaps should be kept off by default until at least a release passes to avoid surprises.
2. Audit data can grow quite large depending on system activity and may blow out the storage system or cause performance issues, etc. There should be some sort of "automatic clean up" process that purges records after a reasonable time period, or sufficient warnings that adopters are to do the cleanup themselves.
--Misagh
----- Original Message -----
> From: "Andrea Patricelli" <an...@apache.org>
> To: "dev" <de...@syncope.apache.org>
> Sent: Tuesday, September 17, 2019 12:02:25 PM
> Subject: [DISCUSS] Object versioning
> Hi all,
>
> in several of the last Syncope projects we recently worked on, we faced
> a recurring issue: data inconsistency (at business level) and the need
> to know who did what on some object.
>
> For "data inconsistency" I do not mean any issue related to database
> records or provisioning, but the values of the user attributes at the
> business level (identity info, groups or resources assigned, etc.).
> Often a sysadmin wonders about "why this attribute has changed?" or "who
> did this (wrong) change, how can I recover the previous value?"
> Changes I'm referring to can come from automatic tasks (pull, scheduled)
> or manual tasks performed by admin or manager users.
> Syncope already provides info about last change date, change password
> date or last modifier user, but in my opinion are not enough to
> understand the history of the record.
>
> This is why I'm proposing to add to Syncope the possibility of
> versioning objects. Something similar has been done for connectors and
> resources configurations [1]. It is not a simple work I know :)
>
> Here are some requirements that I was thinking of:
>
> * Versioning should be done on all objects, USER, GROUP and ANY.
> * Versioning could be heavy for systems with thousands or million of
> users, so it should be a feature to enable/disable at each moment of
> the project lifecycle. It should also be tunable, e.g. store only
> attributes and not resources, groups, etc.
> * In a first release I would leave apart all issues related to
> provisioning. I would simply version data stored on Syncope database
> * User should have the possibility to see a diff of what has changed
> like per connectors and resources configurations.
> * User should have the possibility to restore a certain value (or the
> whole object?)
> * User should be able to see, at least (in a first release), an
> history of what has changed on data (e.g. name: foo -> bar) and who
> did the change and when.
> * User should be able to query by changed data like users who are
> subject to "name" change in a certain period (not so useful, maybe).
>
> There are some usefu l open source libraries that support objects
> versioning like JaVers [2] or [3], but I'm not sure if this last one
> fits all the needs above. If you have any other proposal please don't
> hesitate to attach it in this thread.
>
> WDYT?
>
> Have a nice day,
> Andrea
>
> [1] https://issues.apache.org/jira/browse/SYNCOPE-1145
>
> [2] https://javers.org/
>
> [3]
> https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.auditing
>
> --
> Dott. Andrea Patricelli
> Tel. +39 3204524292
>
> Engineer @ Tirasa S.r.l.
> Viale Vittoria Colonna 97 - 65127 Pescara
> Tel +39 0859116307 / FAX +39 0859111173
> http://www.tirasa.net
>
> Apache Syncope PMC Member