You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Daan Hoogland <da...@gmail.com> on 2015/07/30 14:38:52 UTC
nice write by a Canadian member of our community
http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
spread the word
--
Daan
Re: CloudStack + ELK with log4j-jsonevent-layout
Posted by Santhosh Edukulla <sa...@gmail.com>.
Thomas,
We had a business requirement of correlating logs (sort of SIEM) from
endpoints, edge devices for threat identification. Few of the applications
were in Java with log4j log configuration, few in other streams. I have
pasted the design we followed to configure and forward the logs to elastic
for indexing and analyzing thereafter, look at it. So, we forwarded it to
logstash server, which is configured with log4j encoder, i believe thats
what you want as well, rather than you trying to explicitly converting logs
to json.
http://www.tiikoni.com/tis/view/?id=7a93bfc
Regards,
Santhosh
On Fri, Jul 31, 2015 at 11:03 PM, Thomas Schneider <
thomas.schneider@euskill.com> wrote:
> I downloaded the jar from
>
> http://central.maven.org/maven2/net/logstash/log4j/jsonevent-layout/1.7/jsonevent-layout-1.7.jar
>
> Put it in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/
>
> then edited /etc/cloudstack/management/log4j-cloud.xml
>
> From:
>
> <appender name="FILE"
> class="org.apache.log4j.rolling.RollingFileAppender">
> <param name="Append" value="true"/>
> <param name="Threshold" value="TRACE"/>
> <rollingPolicy
> class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
> <param name="FileNamePattern"
>
> value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
> <param name="ActiveFileName"
> value="/var/log/cloudstack/management/management-server.log"/>
> </rollingPolicy>
> <layout class="org.apache.log4j.EnhancedPatternLayout">
> <param name="ConversionPattern" value="%d{ISO8601} %-5p
> [%c{1.}] (%t:%x) %m%n"/>
> </appender>
>
> To:
> <appender name="FILE"
> class="org.apache.log4j.rolling.RollingFileAppender">
> <param name="Append" value="true"/>
> <param name="Threshold" value="TRACE"/>
> <rollingPolicy
> class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
> <param name="FileNamePattern"
>
> value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
> <param name="ActiveFileName"
> value="/var/log/cloudstack/management/management-server.log"/>
> </rollingPolicy>
> <layout class="net.logstash.log4j.JSONEventLayoutV1" />
> </appender>
>
> but after that I don't have log anymore.
>
> I also tryed to:
> mkdir -p /root/classpath/
> cp jsonevent-layout-1.7.jar /root/classpath
> vi /etc/environement
> add: CLASSPATH="/root/classpath"
> source /etc/environement
>
> but i have the same result.
>
> If someone can advice me ?
>
>
> Le 30/07/2015 21:36, Thomas Schneider a écrit :
> > Hello,
> >
> > I would like to setup ELK stack to monitor CloudStack Log.
> > I have already setup a central Elastisearch + Logstach + Kibana server
> > who receive logs from all my cloudstack management server via
> > logstach-forwader and it work pretty well with the standart system's log
> > file like /var/log/syslog etc... because they can be easyly parsed by
> > logstach's grok filter.
> >
> > But the main problem I have, is I didn't find a good technique to parse
> > cloudstack log file.
> >
> > However i founded a plugin for log4j who's called log4j-jsonevent-layout
> > that can output the cloudstack log in json and the json log are easy to
> > parse for logstache, but a dindn't found how to setup this plugin.
> >
> > So howto setup log4j-jsonevent-layout witch cloudstack ?
> > If someone can advise me on this issue.
> >
> > Regards,
>
> --
> *Thomas Schneider*
> Directeur des Opérations
> Euskill SARL
> Web: www.euskill.com
> Mobile: +33 (0)6 19 26 47 76
> Mail: thomas.schneider@euskill.com
> 5 rue de Phalsbourg
> F-67000 Strasbourg
>
Re: CloudStack + ELK with log4j-jsonevent-layout
Posted by Thomas Schneider <th...@euskill.com>.
I downloaded the jar from
http://central.maven.org/maven2/net/logstash/log4j/jsonevent-layout/1.7/jsonevent-layout-1.7.jar
Put it in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/
then edited /etc/cloudstack/management/log4j-cloud.xml
From:
<appender name="FILE"
class="org.apache.log4j.rolling.RollingFileAppender">
<param name="Append" value="true"/>
<param name="Threshold" value="TRACE"/>
<rollingPolicy
class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern"
value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
<param name="ActiveFileName"
value="/var/log/cloudstack/management/management-server.log"/>
</rollingPolicy>
<layout class="org.apache.log4j.EnhancedPatternLayout">
<param name="ConversionPattern" value="%d{ISO8601} %-5p
[%c{1.}] (%t:%x) %m%n"/>
</appender>
To:
<appender name="FILE"
class="org.apache.log4j.rolling.RollingFileAppender">
<param name="Append" value="true"/>
<param name="Threshold" value="TRACE"/>
<rollingPolicy
class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern"
value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
<param name="ActiveFileName"
value="/var/log/cloudstack/management/management-server.log"/>
</rollingPolicy>
<layout class="net.logstash.log4j.JSONEventLayoutV1" />
</appender>
but after that I don't have log anymore.
I also tryed to:
mkdir -p /root/classpath/
cp jsonevent-layout-1.7.jar /root/classpath
vi /etc/environement
add: CLASSPATH="/root/classpath"
source /etc/environement
but i have the same result.
If someone can advice me ?
Le 30/07/2015 21:36, Thomas Schneider a écrit :
> Hello,
>
> I would like to setup ELK stack to monitor CloudStack Log.
> I have already setup a central Elastisearch + Logstach + Kibana server
> who receive logs from all my cloudstack management server via
> logstach-forwader and it work pretty well with the standart system's log
> file like /var/log/syslog etc... because they can be easyly parsed by
> logstach's grok filter.
>
> But the main problem I have, is I didn't find a good technique to parse
> cloudstack log file.
>
> However i founded a plugin for log4j who's called log4j-jsonevent-layout
> that can output the cloudstack log in json and the json log are easy to
> parse for logstache, but a dindn't found how to setup this plugin.
>
> So howto setup log4j-jsonevent-layout witch cloudstack ?
> If someone can advise me on this issue.
>
> Regards,
--
*Thomas Schneider*
Directeur des Opérations
Euskill SARL
Web: www.euskill.com
Mobile: +33 (0)6 19 26 47 76
Mail: thomas.schneider@euskill.com
5 rue de Phalsbourg
F-67000 Strasbourg
CloudStack + ELK with log4j-jsonevent-layout
Posted by Thomas Schneider <th...@euskill.com>.
Hello,
I would like to setup ELK stack to monitor CloudStack Log.
I have already setup a central Elastisearch + Logstach + Kibana server
who receive logs from all my cloudstack management server via
logstach-forwader and it work pretty well with the standart system's log
file like /var/log/syslog etc... because they can be easyly parsed by
logstach's grok filter.
But the main problem I have, is I didn't find a good technique to parse
cloudstack log file.
However i founded a plugin for log4j who's called log4j-jsonevent-layout
that can output the cloudstack log in json and the json log are easy to
parse for logstache, but a dindn't found how to setup this plugin.
So howto setup log4j-jsonevent-layout witch cloudstack ?
If someone can advise me on this issue.
Regards,
--
*Thomas Schneider*
Re: nice write by a Canadian member of our community
Posted by "Yim, Philip" <Ph...@alliedtelesis.com>.
Hi I would agree with the "blog" sentiments....but there is a rub. In any decision on what to use in the new devops world the decision makers are influenced by their peers and media.
For example at a recent open daylight summit, which is an SDN controller, there was not a single mention of cloudstack only openstack. This may not the audience you want to reach but their influence on the networking industry is significant.
Philip
> On Jul 30, 2015, at 5:39 AM, Daan Hoogland <da...@gmail.com> wrote:
>
> http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
>
> spread the word
> --
> Daan
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator.
Re: nice write by a Canadian member of our community
Posted by Nick Brody <ni...@gmail.com>.
test
On Thu, Jul 30, 2015 at 6:48 AM, Sally Khudairi <sk...@apache.org> wrote:
> Nice write up, indeed! Well done :-)
> From: Daan Hoogland <da...@gmail.com>
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; "
> marketing@cloudstack.apache.org" <ma...@cloudstack.apache.org>
> Sent: Thursday, July 30, 2015 8:38 AM
> Subject: nice write by a Canadian member of our community
>
>
> http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
>
> spread the word
> --
> Daan
>
>
>
>
Re: nice write by a Canadian member of our community
Posted by Sally Khudairi <sk...@apache.org>.
Nice write up, indeed! Well done :-)
From: Daan Hoogland <da...@gmail.com>
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; "marketing@cloudstack.apache.org" <ma...@cloudstack.apache.org>
Sent: Thursday, July 30, 2015 8:38 AM
Subject: nice write by a Canadian member of our community
http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
spread the word
--
Daan
Re: nice write by a Canadian member of our community
Posted by Sally Khudairi <sk...@apache.org>.
Nice write up, indeed! Well done :-)
From: Daan Hoogland <da...@gmail.com>
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; "marketing@cloudstack.apache.org" <ma...@cloudstack.apache.org>
Sent: Thursday, July 30, 2015 8:38 AM
Subject: nice write by a Canadian member of our community
http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
spread the word
--
Daan
Re: nice write by a Canadian member of our community
Posted by Thomas Schneider <th...@euskill.com>.
+1
Le 30/07/2015 15:01, Nick Brody a écrit :
> cool man
>
> On Thu, Jul 30, 2015 at 5:38 AM, Daan Hoogland <da...@gmail.com>
> wrote:
>
>>
>> http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
>>
>> spread the word
>> --
>> Daan
>>
>
--
*Thomas Schneider*
Directeur des Opérations
Euskill SARL
Web: www.euskill.com
Mobile: +33 (0)6 19 26 47 76
Mail: thomas.schneider@euskill.com
5 rue de Phalsbourg
F-67000 Strasbourg
Re: nice write by a Canadian member of our community
Posted by Nick Brody <ni...@gmail.com>.
cool man
On Thu, Jul 30, 2015 at 5:38 AM, Daan Hoogland <da...@gmail.com>
wrote:
>
> http://www.cloudops.com/2015/07/cloudstack-vs-openstack-is-there-really-a-winner/
>
> spread the word
> --
> Daan
>