You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Mohit Durgapal <du...@gmail.com> on 2014/10/27 19:30:51 UTC
flume syslog source max msg size
Hi,
I am using rsyslog to send messages to flume nodes via AWS ELB. On flume
nodes I am using the source type *syslogtcp * where the ELB forwards the
messages. Now I see the messages that are over 2k in size are being broken
into chunks of size 2k when I receive them in flume. As my messages are in
JSON this breaks the structure and my application reading the logs from
hdfs see them as ill-formed JSON and skip those records.
I know that the default message size of rsyslog is 2k. I even increased
that with :
$MaxMessageSize 50k
My rsyslog.conf looks like this:
*# rsyslog v5 configuration file# For more information see
/usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
<http://www.rsyslog.com/doc/troubleshoot.html>#### MODULES ####*
*$MaxMessageSize 50k$ModLoad imuxsock # provides support for local system
logging (e.g. via logger command)$ModLoad imklog # provides kernel
logging support (previously done by rklogd)#$ModLoad immark # provides
--MARK-- message capability# Provides UDP syslog reception#$ModLoad
imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad
imtcp#$InputTCPServerRun 5140#### GLOBAL DIRECTIVES ##### Use default
timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat#
File syncing capability is disabled by default. This feature is usually not
required,# not useful and an extreme performance hit#$ActionFileEnableSync
on# Include all config files in /etc/rsyslog.d/$IncludeConfig
/etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the
console.# Logging much else clutters up the
screen.#kern.*
/dev/console# Log anything (except mail) of level info or higher.# Don't
log private authentication
messages!#*.info;mail.none;authpriv.none;cron.none
/var/log/messages*.info;mail.none;authpriv.none;cron.none;local2.none;local3.none
/var/log/messages# The authpriv file has restricted
access.authpriv.*
/var/log/secure# Log all the mail messages in one
place.mail.*
-/var/log/maillog# Log cron
stuffcron.* /var/log/cron#
Everybody gets emergency
messages*.emerg
:omusrmsg:*# Save news errors of level crit and higher in a special
file.uucp,news.crit
/var/log/spooler# Save boot messages also to
boot.loglocal7.*
/var/log/boot.log$template RTFormat,"%msg%\n"# An on-disk queue is created
for this action. If the remote host is# down, messages are spooled to disk
and sent when it is up again.$WorkDirectory /var/lib/rsyslog # where to
place spool files$ActionQueueFileName fwdRuleRTLogs1 # unique name prefix
for spool files$ActionQueueMaxDiskSpace 5g # 4gb space limit (use as much
as possible)$ActionQueueSaveOnShutdown on # save messages to disk on
shutdown$ActionQueueType LinkedList # run
asynchronously$ActionResumeRetryCount -1 # infinite retries if host is
down# remote host is: name/ip:port, e.g. 192.168.0.1:514
<http://192.168.0.1:514>, port optionallocal2.*
@@internal-load-balancer:5149;RTFormat# ### end of the forwarding rule ###*
Any help would be great.
Regards
Mohit
Re: flume syslog source max msg size
Posted by Mohit Durgapal <du...@gmail.com>.
Hi Jeff & Santiago,
Thanks for your help!! I realized that just after posting that question.
Sorry for not updating it earlier.
Thanks
Mohit
On Thu, Oct 30, 2014 at 5:36 PM, Santiago Mola <sm...@stratio.com> wrote:
> Hi Mohit,
>
> 2014-10-27 19:30 GMT+01:00 Mohit Durgapal <du...@gmail.com>:
>
>>
>> I am using rsyslog to send messages to flume nodes via AWS ELB. On flume
>> nodes I am using the source type *syslogtcp * where the ELB forwards
>> the messages. Now I see the messages that are over 2k in size are being
>> broken into chunks of size 2k when I receive them in flume. As my messages
>> are in JSON this breaks the structure and my application reading the logs
>> from hdfs see them as ill-formed JSON and skip those records.
>>
>
>
> You need to increase the message size both in rsyslog (as you did) *and*
> in your Flume agent.
>
> The Syslog source has an eventSize option that defaults to 2500 bytes. You
> can check the details in the docs:
>
> https://flume.apache.org/FlumeUserGuide.html#syslog-tcp-source
>
> Best,
> --
>
> Santiago M. Mola
>
>
> <http://www.stratio.com/>
> Avenida de Europa, 26. Ática 5. 3ª Planta
> 28224 Pozuelo de Alarcón, Madrid
> Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*
>
Re: flume syslog source max msg size
Posted by Santiago Mola <sm...@stratio.com>.
Hi Mohit,
2014-10-27 19:30 GMT+01:00 Mohit Durgapal <du...@gmail.com>:
>
> I am using rsyslog to send messages to flume nodes via AWS ELB. On flume
> nodes I am using the source type *syslogtcp * where the ELB forwards the
> messages. Now I see the messages that are over 2k in size are being broken
> into chunks of size 2k when I receive them in flume. As my messages are in
> JSON this breaks the structure and my application reading the logs from
> hdfs see them as ill-formed JSON and skip those records.
>
You need to increase the message size both in rsyslog (as you did) *and* in
your Flume agent.
The Syslog source has an eventSize option that defaults to 2500 bytes. You
can check the details in the docs:
https://flume.apache.org/FlumeUserGuide.html#syslog-tcp-source
Best,
--
Santiago M. Mola
<http://www.stratio.com/>
Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*
Re: flume syslog source max msg size
Posted by Jeff Lord <jl...@cloudera.com>.
What about your flume config?
Did you try increasing the eventSize?
On Mon, Oct 27, 2014 at 11:30 AM, Mohit Durgapal <du...@gmail.com>
wrote:
> Hi,
>
> I am using rsyslog to send messages to flume nodes via AWS ELB. On flume
> nodes I am using the source type *syslogtcp * where the ELB forwards the
> messages. Now I see the messages that are over 2k in size are being broken
> into chunks of size 2k when I receive them in flume. As my messages are in
> JSON this breaks the structure and my application reading the logs from
> hdfs see them as ill-formed JSON and skip those records.
>
> I know that the default message size of rsyslog is 2k. I even increased
> that with :
>
> $MaxMessageSize 50k
>
>
> My rsyslog.conf looks like this:
>
>
>
>
>
>
>
>
> *# rsyslog v5 configuration file# For more information see
> /usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see
> http://www.rsyslog.com/doc/troubleshoot.html
> <http://www.rsyslog.com/doc/troubleshoot.html>#### MODULES ####*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *$MaxMessageSize 50k$ModLoad imuxsock # provides support for local system
> logging (e.g. via logger command)$ModLoad imklog # provides kernel
> logging support (previously done by rklogd)#$ModLoad immark # provides
> --MARK-- message capability# Provides UDP syslog reception#$ModLoad
> imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad
> imtcp#$InputTCPServerRun 5140#### GLOBAL DIRECTIVES ##### Use default
> timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat#
> File syncing capability is disabled by default. This feature is usually not
> required,# not useful and an extreme performance hit#$ActionFileEnableSync
> on# Include all config files in /etc/rsyslog.d/$IncludeConfig
> /etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the
> console.# Logging much else clutters up the
> screen.#kern.*
> /dev/console# Log anything (except mail) of level info or higher.# Don't
> log private authentication
> messages!#*.info;mail.none;authpriv.none;cron.none
> /var/log/messages*.info;mail.none;authpriv.none;cron.none;local2.none;local3.none
> /var/log/messages# The authpriv file has restricted
> access.authpriv.*
> /var/log/secure# Log all the mail messages in one
> place.mail.*
> -/var/log/maillog# Log cron
> stuffcron.* /var/log/cron#
> Everybody gets emergency
> messages*.emerg
> :omusrmsg:*# Save news errors of level crit and higher in a special
> file.uucp,news.crit
> /var/log/spooler# Save boot messages also to
> boot.loglocal7.*
> /var/log/boot.log$template RTFormat,"%msg%\n"# An on-disk queue is created
> for this action. If the remote host is# down, messages are spooled to disk
> and sent when it is up again.$WorkDirectory /var/lib/rsyslog # where to
> place spool files$ActionQueueFileName fwdRuleRTLogs1 # unique name prefix
> for spool files$ActionQueueMaxDiskSpace 5g # 4gb space limit (use as much
> as possible)$ActionQueueSaveOnShutdown on # save messages to disk on
> shutdown$ActionQueueType LinkedList # run
> asynchronously$ActionResumeRetryCount -1 # infinite retries if host is
> down# remote host is: name/ip:port, e.g. 192.168.0.1:514
> <http://192.168.0.1:514>, port optionallocal2.*
> @@internal-load-balancer:5149;RTFormat# ### end of the forwarding rule ###*
>
>
>
> Any help would be great.
>
>
> Regards
> Mohit
>