You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@syncope.apache.org by Guido Wimmel <gu...@gmx.net> on 2014/03/27 19:29:54 UTC
cleaning up notification tasks
Hi,
a bit related to the recent discussion of https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number of results stored for Task Exectutions):
When the notification feature is used, each notification generates an entry in the task table, which is thus continually growing.
What would be the usual way to remove old notifications, if necessary? Cleaning up at the database level from time to time?
Cheers,
Guido
Re: Aw: Re: cleaning up notification tasks
Posted by Fabio Martelli <fa...@gmail.com>.
Il 29/03/2014 06:25, Francesco Chicchiriccò ha scritto:
> On 28/03/2014 20:44, Guido Wimmel wrote:
>> Hi Francesco,
>>
>> thanks for the information.
>>
>> As far as I know, for each notification (e.g. on user creation) a
>> notification task is created, and for each notification task there
>> can be a number of task executions.
>>
>> I assume when bulk deleting tasks via the Syncope console, the
>> corresponding task executions are deleted as well?
>
> Exactly.
> Moreover, bulk delete is also available via REST, check [1] and [2].
>
>> What if one e.g. regularly wants to delete all (maybe a large number
>> of) notification tasks that are older than one year?
>> This is currently not possible via the Syncope console and probably
>> not trivial using the REST services, right?
>
> The best way to accomplish this is via a scheduled task [3] (we do
> this in almost all IAM projects we're involved in): basically you can
> leverage this sort of "Java" crontab to perform any operation via DAO
> with Spring transactional support, e.g. exactly as the Syncope core.
Hi, sometimes, for spot tasks, I've chosen for a bash + curl script?
Best regards,
F.
>
> HTH
> Regards.
>
>>> Gesendet: Freitag, 28. März 2014 um 08:39 Uhr
>>> Von: "Francesco Chicchiriccò" <il...@apache.org>
>>> An: user@syncope.apache.org
>>> Betreff: Re: cleaning up notification tasks
>>>
>>> On 27/03/2014 19:29, Guido Wimmel wrote:
>>>> Hi,
>>>>
>>>> a bit related to the recent discussion of
>>>> https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number
>>>> of results stored for Task Exectutions):
>>>>
>>>> When the notification feature is used, each notification generates
>>>> an entry in the task table, which is thus continually growing.
>>>>
>>>> What would be the usual way to remove old notifications, if
>>>> necessary? Cleaning up at the database level from time to time?
>>> Hi Guido,
>>> the easiest way to perform this cleaning is via the admin console, by
>>> leveraging the bulk action feature (leftmost checkbox column, available
>>> in many data tables, then click on the blue gear on the bottom and
>>> choose the delete icon).
>>>
>>> Via REST, task execution removal (for any kind of task: propagation,
>>> synchronization, scheduled, notification) can be performed (see [1] for
>>> a complete reference for 1.1.X) as
>>>
>>> GET /rest/task/execution/delete/{executionId} (Spring MVC)
>>>
>>> or
>>>
>>> DELETE /tasks/executions/{executionId} (CXF, also working in 1.2.X)
>>>
>>> You can also enjoy a sneak peak of REST services documentation in
>>> upcoming 1.2.X by taking a look at [2].
>>>
>>> Consider anyway that if you don't want to store all notification task
>>> executions you can just apply the same workaround suggested in
>>> SYNCOPE-481 for propagation tasks: set notification trace level to
>>> NONE,
>>> FAILURES or SUMMARY instead of ALL (which is the default).
>>>
>>> Hope this helps.
>>> Regards.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/SYNCOPE/REST+API+upgrade#RESTAPIupgrade-TaskService
>>>
>>> [2] http://people.apache.org/~ilgrosso/wadl2html/
> [3]
> https://cwiki.apache.org/confluence/display/SYNCOPE/ScheduledTaskJobClass
>
--
Fabio Martelli
Tirasa - Open Source Excellence
http://www.tirasa.net/
Apache Syncope PMC
http://people.apache.org/~fmartelli/
Re: cleaning up notification tasks
Posted by Guido Wimmel <gu...@gmx.net>.
> Von: "Francesco Chicchiriccò" <il...@apache.org>
> An: user@syncope.apache.org
> Betreff: Re: cleaning up notification tasks
>
> On 29/03/2014 10:00, Guido Wimmel wrote:
> > Hi,
> >
> > ok, this makes sense.
> >
> > I can't see a way to search notification tasks by date, but I assume you could build something based on the ids as well (as Syncope uses sequences).
>
> Generally speaking, tasks don't have timing information: such data is
> found in task executions.
>
> > Scheduled tasks for cleanup might be a nice addition to Syncope core (depending on how project specific they usually are).
>
>
> An idea could be to add
>
> 1. a method to TaskExecDAO for searching task executions (for any
> kind of tasks) completed before a certain date/time
> 2. a scheduled task for removing tasks that only have executions
> completed before a certain date/time
>
> WDYT?
Yes, this should work, but requires extending a Syncope core class.
So, when you said
> >> The best way to accomplish this is via a scheduled task [3] (we do this
> >> in almost all IAM projects we're involved in): [...]
you didn't refer to scheduled tasks in the projects you're involved in that clean up tasks / task executions based
on their age, but to scheduled tasks that clean up tasks / task executions based on some other conditions,
or even to any scheduled tasks in your projects?
Anyway, we don't have an immediate requirement for such clean up tasks, as it will take some time until
database size will become a concern. If it becomes important, we could probably provide a contribution again
(if no one else has done it already then). There might be some caveats, as normal delete operations are very slow
e.g. in Oracle.
Regards,
Guido
Re: cleaning up notification tasks
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 29/03/2014 10:00, Guido Wimmel wrote:
> Hi,
>
> ok, this makes sense.
>
> I can't see a way to search notification tasks by date, but I assume you could build something based on the ids as well (as Syncope uses sequences).
Generally speaking, tasks don't have timing information: such data is
found in task executions.
> Scheduled tasks for cleanup might be a nice addition to Syncope core (depending on how project specific they usually are).
An idea could be to add
1. a method to TaskExecDAO for searching task executions (for any
kind of tasks) completed before a certain date/time
2. a scheduled task for removing tasks that only have executions
completed before a certain date/time
WDYT?
>> Gesendet: Samstag, 29. März 2014 um 06:25 Uhr
>> Von: "Francesco Chicchiriccò" <il...@apache.org>
>> An: user@syncope.apache.org
>> Betreff: Re: Aw: Re: cleaning up notification tasks
>>
>> On 28/03/2014 20:44, Guido Wimmel wrote:
>>> Hi Francesco,
>>>
>>> thanks for the information.
>>>
>>> As far as I know, for each notification (e.g. on user creation) a notification task is created, and for each notification task there can be a number of task executions.
>>>
>>> I assume when bulk deleting tasks via the Syncope console, the corresponding task executions are deleted as well?
>> Exactly.
>> Moreover, bulk delete is also available via REST, check [1] and [2].
>>
>>> What if one e.g. regularly wants to delete all (maybe a large number of) notification tasks that are older than one year?
>>> This is currently not possible via the Syncope console and probably not trivial using the REST services, right?
>> The best way to accomplish this is via a scheduled task [3] (we do this
>> in almost all IAM projects we're involved in): basically you can
>> leverage this sort of "Java" crontab to perform any operation via DAO
>> with Spring transactional support, e.g. exactly as the Syncope core.
>>
>> HTH
>> Regards.
>>
>>>> Gesendet: Freitag, 28. März 2014 um 08:39 Uhr
>>>> Von: "Francesco Chicchiriccò" <il...@apache.org>
>>>> An: user@syncope.apache.org
>>>> Betreff: Re: cleaning up notification tasks
>>>>
>>>> On 27/03/2014 19:29, Guido Wimmel wrote:
>>>>> Hi,
>>>>>
>>>>> a bit related to the recent discussion of https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number of results stored for Task Exectutions):
>>>>>
>>>>> When the notification feature is used, each notification generates an entry in the task table, which is thus continually growing.
>>>>>
>>>>> What would be the usual way to remove old notifications, if necessary? Cleaning up at the database level from time to time?
>>>> Hi Guido,
>>>> the easiest way to perform this cleaning is via the admin console, by
>>>> leveraging the bulk action feature (leftmost checkbox column, available
>>>> in many data tables, then click on the blue gear on the bottom and
>>>> choose the delete icon).
>>>>
>>>> Via REST, task execution removal (for any kind of task: propagation,
>>>> synchronization, scheduled, notification) can be performed (see [1] for
>>>> a complete reference for 1.1.X) as
>>>>
>>>> GET /rest/task/execution/delete/{executionId} (Spring MVC)
>>>>
>>>> or
>>>>
>>>> DELETE /tasks/executions/{executionId} (CXF, also working in 1.2.X)
>>>>
>>>> You can also enjoy a sneak peak of REST services documentation in
>>>> upcoming 1.2.X by taking a look at [2].
>>>>
>>>> Consider anyway that if you don't want to store all notification task
>>>> executions you can just apply the same workaround suggested in
>>>> SYNCOPE-481 for propagation tasks: set notification trace level to NONE,
>>>> FAILURES or SUMMARY instead of ALL (which is the default).
>>>>
>>>> Hope this helps.
>>>> Regards.
>>>>
>>>> [1] https://cwiki.apache.org/confluence/display/SYNCOPE/REST+API+upgrade#RESTAPIupgrade-TaskService
>>>> [2] http://people.apache.org/~ilgrosso/wadl2html/
>> [3] https://cwiki.apache.org/confluence/display/SYNCOPE/ScheduledTaskJobClass
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Involved at The Apache Software Foundation:
member, Syncope PMC chair, Cocoon PMC, Olingo PMC
http://people.apache.org/~ilgrosso/
Aw: Re: Re: cleaning up notification tasks
Posted by Guido Wimmel <gu...@gmx.net>.
Hi,
ok, this makes sense.
I can't see a way to search notification tasks by date, but I assume you could build something based on the ids as well (as Syncope uses sequences).
Scheduled tasks for cleanup might be a nice addition to Syncope core (depending on how project specific they usually are).
Thanks again, Regards,
Guido
> Gesendet: Samstag, 29. März 2014 um 06:25 Uhr
> Von: "Francesco Chicchiriccò" <il...@apache.org>
> An: user@syncope.apache.org
> Betreff: Re: Aw: Re: cleaning up notification tasks
>
> On 28/03/2014 20:44, Guido Wimmel wrote:
> > Hi Francesco,
> >
> > thanks for the information.
> >
> > As far as I know, for each notification (e.g. on user creation) a notification task is created, and for each notification task there can be a number of task executions.
> >
> > I assume when bulk deleting tasks via the Syncope console, the corresponding task executions are deleted as well?
>
> Exactly.
> Moreover, bulk delete is also available via REST, check [1] and [2].
>
> > What if one e.g. regularly wants to delete all (maybe a large number of) notification tasks that are older than one year?
> > This is currently not possible via the Syncope console and probably not trivial using the REST services, right?
>
> The best way to accomplish this is via a scheduled task [3] (we do this
> in almost all IAM projects we're involved in): basically you can
> leverage this sort of "Java" crontab to perform any operation via DAO
> with Spring transactional support, e.g. exactly as the Syncope core.
>
> HTH
> Regards.
>
> >> Gesendet: Freitag, 28. März 2014 um 08:39 Uhr
> >> Von: "Francesco Chicchiriccò" <il...@apache.org>
> >> An: user@syncope.apache.org
> >> Betreff: Re: cleaning up notification tasks
> >>
> >> On 27/03/2014 19:29, Guido Wimmel wrote:
> >>> Hi,
> >>>
> >>> a bit related to the recent discussion of https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number of results stored for Task Exectutions):
> >>>
> >>> When the notification feature is used, each notification generates an entry in the task table, which is thus continually growing.
> >>>
> >>> What would be the usual way to remove old notifications, if necessary? Cleaning up at the database level from time to time?
> >> Hi Guido,
> >> the easiest way to perform this cleaning is via the admin console, by
> >> leveraging the bulk action feature (leftmost checkbox column, available
> >> in many data tables, then click on the blue gear on the bottom and
> >> choose the delete icon).
> >>
> >> Via REST, task execution removal (for any kind of task: propagation,
> >> synchronization, scheduled, notification) can be performed (see [1] for
> >> a complete reference for 1.1.X) as
> >>
> >> GET /rest/task/execution/delete/{executionId} (Spring MVC)
> >>
> >> or
> >>
> >> DELETE /tasks/executions/{executionId} (CXF, also working in 1.2.X)
> >>
> >> You can also enjoy a sneak peak of REST services documentation in
> >> upcoming 1.2.X by taking a look at [2].
> >>
> >> Consider anyway that if you don't want to store all notification task
> >> executions you can just apply the same workaround suggested in
> >> SYNCOPE-481 for propagation tasks: set notification trace level to NONE,
> >> FAILURES or SUMMARY instead of ALL (which is the default).
> >>
> >> Hope this helps.
> >> Regards.
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/SYNCOPE/REST+API+upgrade#RESTAPIupgrade-TaskService
> >> [2] http://people.apache.org/~ilgrosso/wadl2html/
> [3]
> https://cwiki.apache.org/confluence/display/SYNCOPE/ScheduledTaskJobClass
>
> --
> Francesco Chicchiriccò
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Involved at The Apache Software Foundation:
> member, Syncope PMC chair, Cocoon PMC, Olingo PMC
> http://people.apache.org/~ilgrosso/
>
>
Re: Aw: Re: cleaning up notification tasks
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 28/03/2014 20:44, Guido Wimmel wrote:
> Hi Francesco,
>
> thanks for the information.
>
> As far as I know, for each notification (e.g. on user creation) a notification task is created, and for each notification task there can be a number of task executions.
>
> I assume when bulk deleting tasks via the Syncope console, the corresponding task executions are deleted as well?
Exactly.
Moreover, bulk delete is also available via REST, check [1] and [2].
> What if one e.g. regularly wants to delete all (maybe a large number of) notification tasks that are older than one year?
> This is currently not possible via the Syncope console and probably not trivial using the REST services, right?
The best way to accomplish this is via a scheduled task [3] (we do this
in almost all IAM projects we're involved in): basically you can
leverage this sort of "Java" crontab to perform any operation via DAO
with Spring transactional support, e.g. exactly as the Syncope core.
HTH
Regards.
>> Gesendet: Freitag, 28. März 2014 um 08:39 Uhr
>> Von: "Francesco Chicchiriccò" <il...@apache.org>
>> An: user@syncope.apache.org
>> Betreff: Re: cleaning up notification tasks
>>
>> On 27/03/2014 19:29, Guido Wimmel wrote:
>>> Hi,
>>>
>>> a bit related to the recent discussion of https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number of results stored for Task Exectutions):
>>>
>>> When the notification feature is used, each notification generates an entry in the task table, which is thus continually growing.
>>>
>>> What would be the usual way to remove old notifications, if necessary? Cleaning up at the database level from time to time?
>> Hi Guido,
>> the easiest way to perform this cleaning is via the admin console, by
>> leveraging the bulk action feature (leftmost checkbox column, available
>> in many data tables, then click on the blue gear on the bottom and
>> choose the delete icon).
>>
>> Via REST, task execution removal (for any kind of task: propagation,
>> synchronization, scheduled, notification) can be performed (see [1] for
>> a complete reference for 1.1.X) as
>>
>> GET /rest/task/execution/delete/{executionId} (Spring MVC)
>>
>> or
>>
>> DELETE /tasks/executions/{executionId} (CXF, also working in 1.2.X)
>>
>> You can also enjoy a sneak peak of REST services documentation in
>> upcoming 1.2.X by taking a look at [2].
>>
>> Consider anyway that if you don't want to store all notification task
>> executions you can just apply the same workaround suggested in
>> SYNCOPE-481 for propagation tasks: set notification trace level to NONE,
>> FAILURES or SUMMARY instead of ALL (which is the default).
>>
>> Hope this helps.
>> Regards.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/SYNCOPE/REST+API+upgrade#RESTAPIupgrade-TaskService
>> [2] http://people.apache.org/~ilgrosso/wadl2html/
[3]
https://cwiki.apache.org/confluence/display/SYNCOPE/ScheduledTaskJobClass
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Involved at The Apache Software Foundation:
member, Syncope PMC chair, Cocoon PMC, Olingo PMC
http://people.apache.org/~ilgrosso/
Aw: Re: cleaning up notification tasks
Posted by Guido Wimmel <gu...@gmx.net>.
Hi Francesco,
thanks for the information.
As far as I know, for each notification (e.g. on user creation) a notification task is created, and for each notification task there can be a number of task executions.
I assume when bulk deleting tasks via the Syncope console, the corresponding task executions are deleted as well?
What if one e.g. regularly wants to delete all (maybe a large number of) notification tasks that are older than one year?
This is currently not possible via the Syncope console and probably not trivial using the REST services, right?
Best regards,
Guido
> Gesendet: Freitag, 28. März 2014 um 08:39 Uhr
> Von: "Francesco Chicchiriccò" <il...@apache.org>
> An: user@syncope.apache.org
> Betreff: Re: cleaning up notification tasks
>
> On 27/03/2014 19:29, Guido Wimmel wrote:
> > Hi,
> >
> > a bit related to the recent discussion of https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number of results stored for Task Exectutions):
> >
> > When the notification feature is used, each notification generates an entry in the task table, which is thus continually growing.
> >
> > What would be the usual way to remove old notifications, if necessary? Cleaning up at the database level from time to time?
>
> Hi Guido,
> the easiest way to perform this cleaning is via the admin console, by
> leveraging the bulk action feature (leftmost checkbox column, available
> in many data tables, then click on the blue gear on the bottom and
> choose the delete icon).
>
> Via REST, task execution removal (for any kind of task: propagation,
> synchronization, scheduled, notification) can be performed (see [1] for
> a complete reference for 1.1.X) as
>
> GET /rest/task/execution/delete/{executionId} (Spring MVC)
>
> or
>
> DELETE /tasks/executions/{executionId} (CXF, also working in 1.2.X)
>
> You can also enjoy a sneak peak of REST services documentation in
> upcoming 1.2.X by taking a look at [2].
>
> Consider anyway that if you don't want to store all notification task
> executions you can just apply the same workaround suggested in
> SYNCOPE-481 for propagation tasks: set notification trace level to NONE,
> FAILURES or SUMMARY instead of ALL (which is the default).
>
> Hope this helps.
> Regards.
>
> [1]
> https://cwiki.apache.org/confluence/display/SYNCOPE/REST+API+upgrade#RESTAPIupgrade-TaskService
> [2] http://people.apache.org/~ilgrosso/wadl2html/
>
> --
> Francesco Chicchiriccò
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Involved at The Apache Software Foundation:
> member, Syncope PMC chair, Cocoon PMC, Olingo PMC
> http://people.apache.org/~ilgrosso/
>
>
Re: cleaning up notification tasks
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 27/03/2014 19:29, Guido Wimmel wrote:
> Hi,
>
> a bit related to the recent discussion of https://issues.apache.org/jira/browse/SYNCOPE-481 (Limit the number of results stored for Task Exectutions):
>
> When the notification feature is used, each notification generates an entry in the task table, which is thus continually growing.
>
> What would be the usual way to remove old notifications, if necessary? Cleaning up at the database level from time to time?
Hi Guido,
the easiest way to perform this cleaning is via the admin console, by
leveraging the bulk action feature (leftmost checkbox column, available
in many data tables, then click on the blue gear on the bottom and
choose the delete icon).
Via REST, task execution removal (for any kind of task: propagation,
synchronization, scheduled, notification) can be performed (see [1] for
a complete reference for 1.1.X) as
GET /rest/task/execution/delete/{executionId} (Spring MVC)
or
DELETE /tasks/executions/{executionId} (CXF, also working in 1.2.X)
You can also enjoy a sneak peak of REST services documentation in
upcoming 1.2.X by taking a look at [2].
Consider anyway that if you don't want to store all notification task
executions you can just apply the same workaround suggested in
SYNCOPE-481 for propagation tasks: set notification trace level to NONE,
FAILURES or SUMMARY instead of ALL (which is the default).
Hope this helps.
Regards.
[1]
https://cwiki.apache.org/confluence/display/SYNCOPE/REST+API+upgrade#RESTAPIupgrade-TaskService
[2] http://people.apache.org/~ilgrosso/wadl2html/
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Involved at The Apache Software Foundation:
member, Syncope PMC chair, Cocoon PMC, Olingo PMC
http://people.apache.org/~ilgrosso/