You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Sina Kashipazha <es...@yahoo.co.uk.INVALID> on 2020/05/27 13:55:01 UTC
Centralised logging capability.
Centralised logging capability.
Our improvements to systemvm allow Cloudstack administrator to access systemvms logs inside the management server. It removes the difficulty of downloading logs from systemvms. They are forwarded to the management server automatically, and administrators can access them in "/var/log/rsyslog/%HOSTNAME%/syslog”.
It doesn’t require additional work, rsyslog setups on Cloudstack already. The only thing we have to do is open rsyslog port to receive these logs and tell hypervisors and systemvms to forward them to the management server.
Any comments would be highly appreciated.
For more information take a look at below issue and pull request:
Issue: https://github.com/apache/cloudstack/issues/4093
Pull request: https://github.com/apache/cloudstack/pull/4108 <https://github.com/apache/cloudstack/pull/4108>
Kind Regards,
Sina
Re: Centralised logging capability.
Posted by Pierre-Luc Dion <pd...@cloud.ca>.
Hi,
This is a nice idea and initiative, we have a similar project in our
backlog, we cloud help to improve this PR.
There is some issues to send logs to management-server:
* this current feature impose use of rsyslog
* support a single management server , would not work for a redundant
deployment of a management-server
* would require large amount of disk space if log rotate not managed
properly.
* a very good value added for a dev environment
Like Daan is suggesting, if the log destination could be a configurable
parameter, we could change the destination ip and port where to send logs
to a Logstash receiver potentially so we can centralize log in
another system such as an ELK cluster.
On our side we want to implement log forwarding for Virtual-Routers, the
idea we had in mind would be to deploy filebeat in the VR and sent their
log to the hypervisor internal management network, to the hypervisor host,
which would have a Port-Forwarding configure to the log destination IP. So
far it's the simplest way we found it could work relatively safely without
adding a new NIC to VR or interacting with customers side of the VR. We
could also skip filebeat and just forward rsyslog but would need to be UDP
traffic for sure to avoid impacting perf or the VR is the destination is
down or experience high latency.
Cheers,
PL
On Wed, May 27, 2020 at 12:20 PM Sven Vogel <S....@ewerk.com> wrote:
> Hi Sina,
>
> First. Cool feature!
>
> Agree with Daan. This would be absolutely nice to have the possibility to
> send the log files to separate log server. So in the end of the day we have
> an Heavy management server.
>
> Cheers
>
> Sven
>
>
> __
>
> Sven Vogel
> Lead Cloud Solution Architect
>
> EWERK DIGITAL GmbH
> Brühl 24, D-04109 Leipzig
> P +49 341 42649 - 99
> F +49 341 42649 - 98
> S.Vogel@ewerk.com
> www.ewerk.com
>
> Geschäftsführer:
> Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
> Registergericht: Leipzig HRB 9065
>
> Zertifiziert nach:
> ISO/IEC 27001:2013
> DIN EN ISO 9001:2015
> DIN ISO/IEC 20000-1:2011
>
> EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
>
> Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
>
> Disclaimer Privacy:
> Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist
> vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der
> bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
> Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
> informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
> die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System.
> Vielen Dank.
>
> The contents of this e-mail (including any attachments) are confidential
> and may be legally privileged. If you are not the intended recipient of
> this e-mail, any disclosure, copying, distribution or use of its contents
> is strictly prohibited, and you should please notify the sender immediately
> and then delete it (including any attachments) from your system. Thank you.
> > Am 27.05.2020 um 17:43 schrieb Daan Hoogland <da...@gmail.com>:
> >
> > great initiative Sina, I did leave a comment on the PR, about
> > configurability.
> > in short two worries:
> > 1. an operator uses a different log host than the MS (i.e an ip/hostname
> > config)
> > 2. an operator wants to not use the feature (i.e. a boolean flag)
> > I'm not sure how much this would require, but it seems minimal. Of
> course,
> > not opening the port is blocking the feature as well.
> >
> >
> >> On Wed, May 27, 2020 at 3:55 PM Sina Kashipazha
> >> <es...@yahoo.co.uk.invalid> wrote:
> >>
> >>
> >> Centralised logging capability.
> >>
> >> Our improvements to systemvm allow Cloudstack administrator to access
> >> systemvms logs inside the management server. It removes the difficulty
> of
> >> downloading logs from systemvms. They are forwarded to the management
> >> server automatically, and administrators can access them in
> >> "/var/log/rsyslog/%HOSTNAME%/syslog”.
> >>
> >>
> >> It doesn’t require additional work, rsyslog setups on Cloudstack
> already.
> >> The only thing we have to do is open rsyslog port to receive these logs
> and
> >> tell hypervisors and systemvms to forward them to the management server.
> >>
> >> Any comments would be highly appreciated.
> >>
> >> For more information take a look at below issue and pull request:
> >> Issue: https://github.com/apache/cloudstack/issues/4093
> >> Pull request: https://github.com/apache/cloudstack/pull/4108 <
> >> https://github.com/apache/cloudstack/pull/4108>
> >>
> >> Kind Regards,
> >> Sina
> >
> >
> >
> > --
> > Daan
>
Re: Centralised logging capability.
Posted by Sven Vogel <S....@ewerk.com>.
Hi Sina,
First. Cool feature!
Agree with Daan. This would be absolutely nice to have the possibility to send the log files to separate log server. So in the end of the day we have an Heavy management server.
Cheers
Sven
__
Sven Vogel
Lead Cloud Solution Architect
EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 99
F +49 341 42649 - 98
S.Vogel@ewerk.com
www.ewerk.com
Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
Registergericht: Leipzig HRB 9065
Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011
EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.
The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.
> Am 27.05.2020 um 17:43 schrieb Daan Hoogland <da...@gmail.com>:
>
> great initiative Sina, I did leave a comment on the PR, about
> configurability.
> in short two worries:
> 1. an operator uses a different log host than the MS (i.e an ip/hostname
> config)
> 2. an operator wants to not use the feature (i.e. a boolean flag)
> I'm not sure how much this would require, but it seems minimal. Of course,
> not opening the port is blocking the feature as well.
>
>
>> On Wed, May 27, 2020 at 3:55 PM Sina Kashipazha
>> <es...@yahoo.co.uk.invalid> wrote:
>>
>>
>> Centralised logging capability.
>>
>> Our improvements to systemvm allow Cloudstack administrator to access
>> systemvms logs inside the management server. It removes the difficulty of
>> downloading logs from systemvms. They are forwarded to the management
>> server automatically, and administrators can access them in
>> "/var/log/rsyslog/%HOSTNAME%/syslog”.
>>
>>
>> It doesn’t require additional work, rsyslog setups on Cloudstack already.
>> The only thing we have to do is open rsyslog port to receive these logs and
>> tell hypervisors and systemvms to forward them to the management server.
>>
>> Any comments would be highly appreciated.
>>
>> For more information take a look at below issue and pull request:
>> Issue: https://github.com/apache/cloudstack/issues/4093
>> Pull request: https://github.com/apache/cloudstack/pull/4108 <
>> https://github.com/apache/cloudstack/pull/4108>
>>
>> Kind Regards,
>> Sina
>
>
>
> --
> Daan
Re: Centralised logging capability.
Posted by Daan Hoogland <da...@gmail.com>.
great initiative Sina, I did leave a comment on the PR, about
configurability.
in short two worries:
1. an operator uses a different log host than the MS (i.e an ip/hostname
config)
2. an operator wants to not use the feature (i.e. a boolean flag)
I'm not sure how much this would require, but it seems minimal. Of course,
not opening the port is blocking the feature as well.
On Wed, May 27, 2020 at 3:55 PM Sina Kashipazha
<es...@yahoo.co.uk.invalid> wrote:
>
> Centralised logging capability.
>
> Our improvements to systemvm allow Cloudstack administrator to access
> systemvms logs inside the management server. It removes the difficulty of
> downloading logs from systemvms. They are forwarded to the management
> server automatically, and administrators can access them in
> "/var/log/rsyslog/%HOSTNAME%/syslog”.
>
>
> It doesn’t require additional work, rsyslog setups on Cloudstack already.
> The only thing we have to do is open rsyslog port to receive these logs and
> tell hypervisors and systemvms to forward them to the management server.
>
> Any comments would be highly appreciated.
>
> For more information take a look at below issue and pull request:
> Issue: https://github.com/apache/cloudstack/issues/4093
> Pull request: https://github.com/apache/cloudstack/pull/4108 <
> https://github.com/apache/cloudstack/pull/4108>
>
> Kind Regards,
> Sina
--
Daan