You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@openmeetings.apache.org by Thibault Le Meur <Th...@supelec.fr> on 2012/04/06 12:28:06 UTC
Shell script to delete unsused streams and report stream diskspace
usage
Hi,
It seems that OM doesn't delete streams form the filesystem. Thus I
needed a way to track diskspace usage of streams so I developped a "not
so quick and yet dirty script" to do the following tasks:
* delete streams marked as deleted in OpenMeetings DB from the filesystem
* report diskspace used by not deleted streams with associated
information (owner, room, ...)
* report diskspace per user, and send an overquota email if needed
* report streams older than and Expiration duration threshold and send
an expriation email if needed
It requires:
* linux system
* openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707)
* mysql client, a mysqk user able to do selects to the OM DB
* usual linux tools like cut, awk, sed and bash 4.x
It comes with no warranty as I had to reverse engineer the DB to
understand how streams ar recorded and marked as deleted and I'm quite
sure there are things I missed so It may be considered as a "template".
I just post it here, in case it can prove to be usefull to someone.
Now I would need help on how uploaded files are recorded to filesystem
and DB to do the same for these ones. If anyone can help me... ;-)
Regards,
Thibault
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
I have added this functionality to ./admin.sh (admin.bat) (It uses
java calls and should work on all systems, with all databases)
On Fri, Apr 6, 2012 at 11:13 PM, Maxim Solodovnik <so...@gmail.com> wrote:
> Thanks for the idea :)
>
>
> On Fri, Apr 6, 2012 at 23:11, <t....@gmail.com> wrote:
>>
>> Le 06/04/2012 18:05, Maxim Solodovnik a écrit :
>> > I will add multi-OS and multi-DB version of this.
>> > I guess having cleanup capabilities is vital while storing video files
>> > ...
>> >
>>
>> Ok.
>>
>> I haven't replied to the licensing question.
>> I see no issue in licensing my own work as ASF, however I used some
>> cut/paste code taken from webpages on the Net (essentially the
>> bytesToHuman and isNotSet functions), so I'm not sure if I can impose an
>> ASF licence to it.
>>
>> Thibault
>>
>
>
>
> --
> WBR
> Maxim aka solomax
--
WBR
Maxim aka solomax
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
I have added this functionality to ./admin.sh (admin.bat) (It uses
java calls and should work on all systems, with all databases)
On Fri, Apr 6, 2012 at 11:13 PM, Maxim Solodovnik <so...@gmail.com> wrote:
> Thanks for the idea :)
>
>
> On Fri, Apr 6, 2012 at 23:11, <t....@gmail.com> wrote:
>>
>> Le 06/04/2012 18:05, Maxim Solodovnik a écrit :
>> > I will add multi-OS and multi-DB version of this.
>> > I guess having cleanup capabilities is vital while storing video files
>> > ...
>> >
>>
>> Ok.
>>
>> I haven't replied to the licensing question.
>> I see no issue in licensing my own work as ASF, however I used some
>> cut/paste code taken from webpages on the Net (essentially the
>> bytesToHuman and isNotSet functions), so I'm not sure if I can impose an
>> ASF licence to it.
>>
>> Thibault
>>
>
>
>
> --
> WBR
> Maxim aka solomax
--
WBR
Maxim aka solomax
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
Thanks for the idea :)
On Fri, Apr 6, 2012 at 23:11, <t....@gmail.com> wrote:
> Le 06/04/2012 18:05, Maxim Solodovnik a écrit :
> > I will add multi-OS and multi-DB version of this.
> > I guess having cleanup capabilities is vital while storing video files
> ...
> >
>
> Ok.
>
> I haven't replied to the licensing question.
> I see no issue in licensing my own work as ASF, however I used some
> cut/paste code taken from webpages on the Net (essentially the
> bytesToHuman and isNotSet functions), so I'm not sure if I can impose an
> ASF licence to it.
>
> Thibault
>
>
--
WBR
Maxim aka solomax
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
Thanks for the idea :)
On Fri, Apr 6, 2012 at 23:11, <t....@gmail.com> wrote:
> Le 06/04/2012 18:05, Maxim Solodovnik a écrit :
> > I will add multi-OS and multi-DB version of this.
> > I guess having cleanup capabilities is vital while storing video files
> ...
> >
>
> Ok.
>
> I haven't replied to the licensing question.
> I see no issue in licensing my own work as ASF, however I used some
> cut/paste code taken from webpages on the Net (essentially the
> bytesToHuman and isNotSet functions), so I'm not sure if I can impose an
> ASF licence to it.
>
> Thibault
>
>
--
WBR
Maxim aka solomax
Re: Shell script to delete unsused streams and report stream diskspace
usage
Posted by t....@gmail.com.
Le 06/04/2012 18:05, Maxim Solodovnik a écrit :
> I will add multi-OS and multi-DB version of this.
> I guess having cleanup capabilities is vital while storing video files ...
>
Ok.
I haven't replied to the licensing question.
I see no issue in licensing my own work as ASF, however I used some
cut/paste code taken from webpages on the Net (essentially the
bytesToHuman and isNotSet functions), so I'm not sure if I can impose an
ASF licence to it.
Thibault
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
I will add multi-OS and multi-DB version of this.
I guess having cleanup capabilities is vital while storing video files ...
On Fri, Apr 6, 2012 at 22:58, Thibault Le Meur
<Th...@supelec.fr>wrote:
> Hi Alexei,
>
> I knopw about jira contributions, but I don't think this should be
> included into the OM disrtibutions.
> Looks more like a "tip and tricks" item, you know, things you put in a
> wiki, than a script to be added to the distribution (as I've said it's
> quick and dirty).
> It can be used until OM has it's own clean way of handling these
> operations (User quota, file expirations, ...).
>
> By the way, here is the little brother of the first script, aimed at
> dealing with files uploaded to rooms.
>
> Regards,
> Thibault
>
> Le 06/04/2012 12:56, Alexei Fedotov a écrit :
> > Thibault,
> >
> > thanks for your contribution. The recommended way is to contribute via
> > jira attachment, in that case you transfer rights to ASF correctly.
> >
> > --
> > With best regards / с наилучшими пожеланиями,
> > Alexei Fedotov / Алексей Федотов,
> > http://dataved.ru/
> > +7 916 562 8095
> >
> >
> >
> > On Fri, Apr 6, 2012 at 2:28 PM, Thibault Le Meur
> > <Th...@supelec.fr> wrote:
> >> Hi,
> >>
> >> It seems that OM doesn't delete streams form the filesystem. Thus I
> >> needed a way to track diskspace usage of streams so I developped a "not
> >> so quick and yet dirty script" to do the following tasks:
> >> * delete streams marked as deleted in OpenMeetings DB from the
> filesystem
> >> * report diskspace used by not deleted streams with associated
> >> information (owner, room, ...)
> >> * report diskspace per user, and send an overquota email if needed
> >> * report streams older than and Expiration duration threshold and send
> >> an expriation email if needed
> >>
> >> It requires:
> >> * linux system
> >> * openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707)
> >> * mysql client, a mysqk user able to do selects to the OM DB
> >> * usual linux tools like cut, awk, sed and bash 4.x
> >>
> >> It comes with no warranty as I had to reverse engineer the DB to
> >> understand how streams ar recorded and marked as deleted and I'm quite
> >> sure there are things I missed so It may be considered as a "template".
> >> I just post it here, in case it can prove to be usefull to someone.
> >>
> >> Now I would need help on how uploaded files are recorded to filesystem
> >> and DB to do the same for these ones. If anyone can help me... ;-)
> >>
> >> Regards,
> >> Thibault
>
>
--
WBR
Maxim aka solomax
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
I will add multi-OS and multi-DB version of this.
I guess having cleanup capabilities is vital while storing video files ...
On Fri, Apr 6, 2012 at 22:58, Thibault Le Meur
<Th...@supelec.fr>wrote:
> Hi Alexei,
>
> I knopw about jira contributions, but I don't think this should be
> included into the OM disrtibutions.
> Looks more like a "tip and tricks" item, you know, things you put in a
> wiki, than a script to be added to the distribution (as I've said it's
> quick and dirty).
> It can be used until OM has it's own clean way of handling these
> operations (User quota, file expirations, ...).
>
> By the way, here is the little brother of the first script, aimed at
> dealing with files uploaded to rooms.
>
> Regards,
> Thibault
>
> Le 06/04/2012 12:56, Alexei Fedotov a écrit :
> > Thibault,
> >
> > thanks for your contribution. The recommended way is to contribute via
> > jira attachment, in that case you transfer rights to ASF correctly.
> >
> > --
> > With best regards / с наилучшими пожеланиями,
> > Alexei Fedotov / Алексей Федотов,
> > http://dataved.ru/
> > +7 916 562 8095
> >
> >
> >
> > On Fri, Apr 6, 2012 at 2:28 PM, Thibault Le Meur
> > <Th...@supelec.fr> wrote:
> >> Hi,
> >>
> >> It seems that OM doesn't delete streams form the filesystem. Thus I
> >> needed a way to track diskspace usage of streams so I developped a "not
> >> so quick and yet dirty script" to do the following tasks:
> >> * delete streams marked as deleted in OpenMeetings DB from the
> filesystem
> >> * report diskspace used by not deleted streams with associated
> >> information (owner, room, ...)
> >> * report diskspace per user, and send an overquota email if needed
> >> * report streams older than and Expiration duration threshold and send
> >> an expriation email if needed
> >>
> >> It requires:
> >> * linux system
> >> * openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707)
> >> * mysql client, a mysqk user able to do selects to the OM DB
> >> * usual linux tools like cut, awk, sed and bash 4.x
> >>
> >> It comes with no warranty as I had to reverse engineer the DB to
> >> understand how streams ar recorded and marked as deleted and I'm quite
> >> sure there are things I missed so It may be considered as a "template".
> >> I just post it here, in case it can prove to be usefull to someone.
> >>
> >> Now I would need help on how uploaded files are recorded to filesystem
> >> and DB to do the same for these ones. If anyone can help me... ;-)
> >>
> >> Regards,
> >> Thibault
>
>
--
WBR
Maxim aka solomax
Re: Shell script to delete unsused streams and report stream diskspace
usage
Posted by Thibault Le Meur <Th...@supelec.fr>.
Hi Alexei,
I knopw about jira contributions, but I don't think this should be
included into the OM disrtibutions.
Looks more like a "tip and tricks" item, you know, things you put in a
wiki, than a script to be added to the distribution (as I've said it's
quick and dirty).
It can be used until OM has it's own clean way of handling these
operations (User quota, file expirations, ...).
By the way, here is the little brother of the first script, aimed at
dealing with files uploaded to rooms.
Regards,
Thibault
Le 06/04/2012 12:56, Alexei Fedotov a écrit :
> Thibault,
>
> thanks for your contribution. The recommended way is to contribute via
> jira attachment, in that case you transfer rights to ASF correctly.
>
> --
> With best regards / с наилучшими пожеланиями,
> Alexei Fedotov / Алексей Федотов,
> http://dataved.ru/
> +7 916 562 8095
>
>
>
> On Fri, Apr 6, 2012 at 2:28 PM, Thibault Le Meur
> <Th...@supelec.fr> wrote:
>> Hi,
>>
>> It seems that OM doesn't delete streams form the filesystem. Thus I
>> needed a way to track diskspace usage of streams so I developped a "not
>> so quick and yet dirty script" to do the following tasks:
>> * delete streams marked as deleted in OpenMeetings DB from the filesystem
>> * report diskspace used by not deleted streams with associated
>> information (owner, room, ...)
>> * report diskspace per user, and send an overquota email if needed
>> * report streams older than and Expiration duration threshold and send
>> an expriation email if needed
>>
>> It requires:
>> * linux system
>> * openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707)
>> * mysql client, a mysqk user able to do selects to the OM DB
>> * usual linux tools like cut, awk, sed and bash 4.x
>>
>> It comes with no warranty as I had to reverse engineer the DB to
>> understand how streams ar recorded and marked as deleted and I'm quite
>> sure there are things I missed so It may be considered as a "template".
>> I just post it here, in case it can prove to be usefull to someone.
>>
>> Now I would need help on how uploaded files are recorded to filesystem
>> and DB to do the same for these ones. If anyone can help me... ;-)
>>
>> Regards,
>> Thibault
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Alexei Fedotov <al...@gmail.com>.
Thibault,
thanks for your contribution. The recommended way is to contribute via
jira attachment, in that case you transfer rights to ASF correctly.
--
With best regards / с наилучшими пожеланиями,
Alexei Fedotov / Алексей Федотов,
http://dataved.ru/
+7 916 562 8095
On Fri, Apr 6, 2012 at 2:28 PM, Thibault Le Meur
<Th...@supelec.fr> wrote:
> Hi,
>
> It seems that OM doesn't delete streams form the filesystem. Thus I
> needed a way to track diskspace usage of streams so I developped a "not
> so quick and yet dirty script" to do the following tasks:
> * delete streams marked as deleted in OpenMeetings DB from the filesystem
> * report diskspace used by not deleted streams with associated
> information (owner, room, ...)
> * report diskspace per user, and send an overquota email if needed
> * report streams older than and Expiration duration threshold and send
> an expriation email if needed
>
> It requires:
> * linux system
> * openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707)
> * mysql client, a mysqk user able to do selects to the OM DB
> * usual linux tools like cut, awk, sed and bash 4.x
>
> It comes with no warranty as I had to reverse engineer the DB to
> understand how streams ar recorded and marked as deleted and I'm quite
> sure there are things I missed so It may be considered as a "template".
> I just post it here, in case it can prove to be usefull to someone.
>
> Now I would need help on how uploaded files are recorded to filesystem
> and DB to do the same for these ones. If anyone can help me... ;-)
>
> Regards,
> Thibault
Re: Shell script to delete unsused streams and report stream
diskspace usage
Posted by Maxim Solodovnik <so...@gmail.com>.
Hello Thibault,
Rhanks for your effort.
I'll take a look at this script. As an alternative to the reverse
engineering I can propose command line tool using Java.
Or maybe extend current command line admin (OPENMEETINGS-111)
On Apr 6, 2012 5:28 PM, "Thibault Le Meur" <Th...@supelec.fr>
wrote:
> Hi,
>
> It seems that OM doesn't delete streams form the filesystem. Thus I
> needed a way to track diskspace usage of streams so I developped a "not
> so quick and yet dirty script" to do the following tasks:
> * delete streams marked as deleted in OpenMeetings DB from the filesystem
> * report diskspace used by not deleted streams with associated
> information (owner, room, ...)
> * report diskspace per user, and send an overquota email if needed
> * report streams older than and Expiration duration threshold and send
> an expriation email if needed
>
> It requires:
> * linux system
> * openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707)
> * mysql client, a mysqk user able to do selects to the OM DB
> * usual linux tools like cut, awk, sed and bash 4.x
>
> It comes with no warranty as I had to reverse engineer the DB to
> understand how streams ar recorded and marked as deleted and I'm quite
> sure there are things I missed so It may be considered as a "template".
> I just post it here, in case it can prove to be usefull to someone.
>
> Now I would need help on how uploaded files are recorded to filesystem
> and DB to do the same for these ones. If anyone can help me... ;-)
>
> Regards,
> Thibault
>