You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Thomas Schneider <th...@euskill.com> on 2015/07/30 21:36:57 UTC
CloudStack + ELK with log4j-jsonevent-layout
Hello,
I would like to setup ELK stack to monitor CloudStack Log.
I have already setup a central Elastisearch + Logstach + Kibana server
who receive logs from all my cloudstack management server via
logstach-forwader and it work pretty well with the standart system's log
file like /var/log/syslog etc... because they can be easyly parsed by
logstach's grok filter.
But the main problem I have, is I didn't find a good technique to parse
cloudstack log file.
However i founded a plugin for log4j who's called log4j-jsonevent-layout
that can output the cloudstack log in json and the json log are easy to
parse for logstache, but a dindn't found how to setup this plugin.
So howto setup log4j-jsonevent-layout witch cloudstack ?
If someone can advise me on this issue.
Regards,
--
*Thomas Schneider*
Re: CloudStack + ELK with log4j-jsonevent-layout
Posted by Santhosh Edukulla <sa...@gmail.com>.
Thomas,
We had a business requirement of correlating logs (sort of SIEM) from
endpoints, edge devices for threat identification. Few of the applications
were in Java with log4j log configuration, few in other streams. I have
pasted the design we followed to configure and forward the logs to elastic
for indexing and analyzing thereafter, look at it. So, we forwarded it to
logstash server, which is configured with log4j encoder, i believe thats
what you want as well, rather than you trying to explicitly converting logs
to json.
http://www.tiikoni.com/tis/view/?id=7a93bfc
Regards,
Santhosh
On Fri, Jul 31, 2015 at 11:03 PM, Thomas Schneider <
thomas.schneider@euskill.com> wrote:
> I downloaded the jar from
>
> http://central.maven.org/maven2/net/logstash/log4j/jsonevent-layout/1.7/jsonevent-layout-1.7.jar
>
> Put it in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/
>
> then edited /etc/cloudstack/management/log4j-cloud.xml
>
> From:
>
> <appender name="FILE"
> class="org.apache.log4j.rolling.RollingFileAppender">
> <param name="Append" value="true"/>
> <param name="Threshold" value="TRACE"/>
> <rollingPolicy
> class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
> <param name="FileNamePattern"
>
> value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
> <param name="ActiveFileName"
> value="/var/log/cloudstack/management/management-server.log"/>
> </rollingPolicy>
> <layout class="org.apache.log4j.EnhancedPatternLayout">
> <param name="ConversionPattern" value="%d{ISO8601} %-5p
> [%c{1.}] (%t:%x) %m%n"/>
> </appender>
>
> To:
> <appender name="FILE"
> class="org.apache.log4j.rolling.RollingFileAppender">
> <param name="Append" value="true"/>
> <param name="Threshold" value="TRACE"/>
> <rollingPolicy
> class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
> <param name="FileNamePattern"
>
> value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
> <param name="ActiveFileName"
> value="/var/log/cloudstack/management/management-server.log"/>
> </rollingPolicy>
> <layout class="net.logstash.log4j.JSONEventLayoutV1" />
> </appender>
>
> but after that I don't have log anymore.
>
> I also tryed to:
> mkdir -p /root/classpath/
> cp jsonevent-layout-1.7.jar /root/classpath
> vi /etc/environement
> add: CLASSPATH="/root/classpath"
> source /etc/environement
>
> but i have the same result.
>
> If someone can advice me ?
>
>
> Le 30/07/2015 21:36, Thomas Schneider a écrit :
> > Hello,
> >
> > I would like to setup ELK stack to monitor CloudStack Log.
> > I have already setup a central Elastisearch + Logstach + Kibana server
> > who receive logs from all my cloudstack management server via
> > logstach-forwader and it work pretty well with the standart system's log
> > file like /var/log/syslog etc... because they can be easyly parsed by
> > logstach's grok filter.
> >
> > But the main problem I have, is I didn't find a good technique to parse
> > cloudstack log file.
> >
> > However i founded a plugin for log4j who's called log4j-jsonevent-layout
> > that can output the cloudstack log in json and the json log are easy to
> > parse for logstache, but a dindn't found how to setup this plugin.
> >
> > So howto setup log4j-jsonevent-layout witch cloudstack ?
> > If someone can advise me on this issue.
> >
> > Regards,
>
> --
> *Thomas Schneider*
> Directeur des Opérations
> Euskill SARL
> Web: www.euskill.com
> Mobile: +33 (0)6 19 26 47 76
> Mail: thomas.schneider@euskill.com
> 5 rue de Phalsbourg
> F-67000 Strasbourg
>
Re: CloudStack + ELK with log4j-jsonevent-layout
Posted by Thomas Schneider <th...@euskill.com>.
I downloaded the jar from
http://central.maven.org/maven2/net/logstash/log4j/jsonevent-layout/1.7/jsonevent-layout-1.7.jar
Put it in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/
then edited /etc/cloudstack/management/log4j-cloud.xml
From:
<appender name="FILE"
class="org.apache.log4j.rolling.RollingFileAppender">
<param name="Append" value="true"/>
<param name="Threshold" value="TRACE"/>
<rollingPolicy
class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern"
value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
<param name="ActiveFileName"
value="/var/log/cloudstack/management/management-server.log"/>
</rollingPolicy>
<layout class="org.apache.log4j.EnhancedPatternLayout">
<param name="ConversionPattern" value="%d{ISO8601} %-5p
[%c{1.}] (%t:%x) %m%n"/>
</appender>
To:
<appender name="FILE"
class="org.apache.log4j.rolling.RollingFileAppender">
<param name="Append" value="true"/>
<param name="Threshold" value="TRACE"/>
<rollingPolicy
class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern"
value="/var/log/cloudstack/management/management-server.log.%d{yyyy-MM-dd}.gz"/>
<param name="ActiveFileName"
value="/var/log/cloudstack/management/management-server.log"/>
</rollingPolicy>
<layout class="net.logstash.log4j.JSONEventLayoutV1" />
</appender>
but after that I don't have log anymore.
I also tryed to:
mkdir -p /root/classpath/
cp jsonevent-layout-1.7.jar /root/classpath
vi /etc/environement
add: CLASSPATH="/root/classpath"
source /etc/environement
but i have the same result.
If someone can advice me ?
Le 30/07/2015 21:36, Thomas Schneider a écrit :
> Hello,
>
> I would like to setup ELK stack to monitor CloudStack Log.
> I have already setup a central Elastisearch + Logstach + Kibana server
> who receive logs from all my cloudstack management server via
> logstach-forwader and it work pretty well with the standart system's log
> file like /var/log/syslog etc... because they can be easyly parsed by
> logstach's grok filter.
>
> But the main problem I have, is I didn't find a good technique to parse
> cloudstack log file.
>
> However i founded a plugin for log4j who's called log4j-jsonevent-layout
> that can output the cloudstack log in json and the json log are easy to
> parse for logstache, but a dindn't found how to setup this plugin.
>
> So howto setup log4j-jsonevent-layout witch cloudstack ?
> If someone can advise me on this issue.
>
> Regards,
--
*Thomas Schneider*
Directeur des Opérations
Euskill SARL
Web: www.euskill.com
Mobile: +33 (0)6 19 26 47 76
Mail: thomas.schneider@euskill.com
5 rue de Phalsbourg
F-67000 Strasbourg