You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by mete <ef...@gmail.com> on 2012/07/25 09:49:54 UTC
flume-ng & syslog source
Hello folks,
I am using flume-ng for cdh4 (1.10), and i am redirecting syslog output
from a network device to flume-ng. My config is as follows:
test1.channels.mem-chan-1.type = memory
test1.channels.mem-chan-1.capacity = 100000
test1.channels.mem-chan-1.transactionCapacity = 1000
test1.sources.syslog-traffic.channels = mem-chan-1
test1.sources.syslog-traffic.type = syslogudp
test1.sources.syslog-traffic.port = 5140
test1.sources.syslog-traffic.bind = test1
test1.sources.syslog-traffic.eventSize = 10000
test1.sinks.file-sink-1.channel = mem-chan-1
test1.sinks.file-sink-1.type = file_roll
test1.sinks.file-sink-1.sink.directory = /home/cloudera-user/tmp/
test1.sinks.file-sink-1.rollInterval = 86400
test1.channels = mem-chan-1
test1.sources = syslog-traffic
test1.sinks = file-sink-1
i have a pretty straightforward config with one syslogudp source , a memory
channel and a file sink.
However, some of the messages i see on the file is like this:
DEVICE: "some syslog content"@
DEVICE: "some syslog content"@
OUT^@
FIN^@
RST^@
RST^@
OUT^@
FIN^@
RST^@
FIN^@
FIN^@
OUT^@
RST^@
RST^@
RST^@
As you can see, some lines are somehow trimmed and does not contain the
entire message. When i redirect same device to syslog-ng there are no
issues like this.
I tried increasing the event size on the syslog source but that did not
change anything at all.
Any ideas on what might be the problem?
Thanks in advance.
Mete
Re: flume-ng & syslog source
Posted by mete <ef...@gmail.com>.
sorry to bump this old one but i saw something interesting with dashes,
when i send a syslog message like the following:
*"my string - with - dashes inside - of it"*
flume output is like
*"- dashes inside - of it"*
in the SyslogUDPSource class, when the event is extracted it loses some
part of the message..
Any ideas?
Regards
On Wed, Jul 25, 2012 at 12:50 PM, mete <ef...@gmail.com> wrote:
> Hello Hari,
>
> I tried to correlate two logs and here is a sample event:
>
> *Http Connection Event On Syslog NG:*
>
> SSG550: NetScreen device_id=SSG550
> [Root]system-notification-00257(traffic): start_time="2012-07-11 19:06:51"
> duration=0 policy_id=10 service=http proto=6 src zone=Trust dst zone=Trust
> action=Permit sent=0 rcvd=0 src=IP1 dst=IP2 src_port=57829 dst_port=80
> src-xlated ip=IP1 port=57829 dst-xlated ip=IP2 port=80 session_id=254877 *
> reason=Creation*
>
> SSG550: NetScreen device_id=SSG550
> [Root]system-notification-00257(traffic): start_time="2012-07-11 19:06:51"
> duration=14 policy_id=10 service=http proto=6 src zone=Trust dst zone=Trust
> action=Permit sent=2402 rcvd=8364 src=IP1 dst=IP2 src_port=57829
> dst_port=80 src-xlated ip=IP1 port=57829 dst-xlated ip=IP2 port=80
> session_id=254877 *reason=Close - TCP FIN*
>
>
>
> *Same Event on Flume:*
>
> SSG550: NetScreen device_id=SSG550
> [Root]system-notification-00257(traffic): start_time="2012-07-11 19:06:51"
> duration=0 policy_id=10 service=http proto=6 src zone=Trust dst zone=Trust
> action=Permit sent=0 rcvd=0 src=IP1 dst=IP2 src_port=57829 dst_port=80
> src-xlated ip=IP1 port=57829 dst-xlated ip=IP2 port=80 session_id=254877 *
> reason=Creation^@*
>
> FIN^@
>
> In general, on the flume side, i cannot see any close,age out,finish logs
> properly. For this event, i cannot find an end event, and i assumed one of
> the FIN message belongs to that. For the end events i see lines like:
>
> FIN^@
> FIN^@
> Unreach^@
> FIN^@
> OUT^@
> OUT^@
> FIN^@
> RST^@
>
> Any ideas?
> Thanks in advance.
>
> Mete
>
>
>
>
>
>
> On Wed, Jul 25, 2012 at 10:55 AM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>> It would be helpful if you could send the original messages as well.
>>
>> Thanks
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>> On Wednesday, July 25, 2012 at 12:49 AM, mete wrote:
>>
>> Hello folks,
>>
>> I am using flume-ng for cdh4 (1.10), and i am redirecting syslog output
>> from a network device to flume-ng. My config is as follows:
>> test1.channels.mem-chan-1.type = memory
>> test1.channels.mem-chan-1.capacity = 100000
>> test1.channels.mem-chan-1.transactionCapacity = 1000
>>
>> test1.sources.syslog-traffic.channels = mem-chan-1
>> test1.sources.syslog-traffic.type = syslogudp
>> test1.sources.syslog-traffic.port = 5140
>> test1.sources.syslog-traffic.bind = test1
>> test1.sources.syslog-traffic.eventSize = 10000
>>
>> test1.sinks.file-sink-1.channel = mem-chan-1
>> test1.sinks.file-sink-1.type = file_roll
>> test1.sinks.file-sink-1.sink.directory = /home/cloudera-user/tmp/
>> test1.sinks.file-sink-1.rollInterval = 86400
>>
>> test1.channels = mem-chan-1
>> test1.sources = syslog-traffic
>> test1.sinks = file-sink-1
>>
>> i have a pretty straightforward config with one syslogudp source , a
>> memory channel and a file sink.
>>
>> However, some of the messages i see on the file is like this:
>>
>> DEVICE: "some syslog content"@
>> DEVICE: "some syslog content"@
>> OUT^@
>> FIN^@
>> RST^@
>> RST^@
>> OUT^@
>> FIN^@
>> RST^@
>> FIN^@
>> FIN^@
>> OUT^@
>> RST^@
>> RST^@
>> RST^@
>>
>> As you can see, some lines are somehow trimmed and does not contain the
>> entire message. When i redirect same device to syslog-ng there are no
>> issues like this.
>> I tried increasing the event size on the syslog source but that did not
>> change anything at all.
>> Any ideas on what might be the problem?
>> Thanks in advance.
>>
>> Mete
>>
>>
>>
>
Re: flume-ng & syslog source
Posted by mete <ef...@gmail.com>.
Hello Hari,
I tried to correlate two logs and here is a sample event:
*Http Connection Event On Syslog NG:*
SSG550: NetScreen device_id=SSG550
[Root]system-notification-00257(traffic): start_time="2012-07-11 19:06:51"
duration=0 policy_id=10 service=http proto=6 src zone=Trust dst zone=Trust
action=Permit sent=0 rcvd=0 src=IP1 dst=IP2 src_port=57829 dst_port=80
src-xlated ip=IP1 port=57829 dst-xlated ip=IP2 port=80 session_id=254877 *
reason=Creation*
SSG550: NetScreen device_id=SSG550
[Root]system-notification-00257(traffic): start_time="2012-07-11 19:06:51"
duration=14 policy_id=10 service=http proto=6 src zone=Trust dst zone=Trust
action=Permit sent=2402 rcvd=8364 src=IP1 dst=IP2 src_port=57829
dst_port=80 src-xlated ip=IP1 port=57829 dst-xlated ip=IP2 port=80
session_id=254877 *reason=Close - TCP FIN*
*Same Event on Flume:*
SSG550: NetScreen device_id=SSG550
[Root]system-notification-00257(traffic): start_time="2012-07-11 19:06:51"
duration=0 policy_id=10 service=http proto=6 src zone=Trust dst zone=Trust
action=Permit sent=0 rcvd=0 src=IP1 dst=IP2 src_port=57829 dst_port=80
src-xlated ip=IP1 port=57829 dst-xlated ip=IP2 port=80 session_id=254877 *
reason=Creation^@*
FIN^@
In general, on the flume side, i cannot see any close,age out,finish logs
properly. For this event, i cannot find an end event, and i assumed one of
the FIN message belongs to that. For the end events i see lines like:
FIN^@
FIN^@
Unreach^@
FIN^@
OUT^@
OUT^@
FIN^@
RST^@
Any ideas?
Thanks in advance.
Mete
On Wed, Jul 25, 2012 at 10:55 AM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:
> It would be helpful if you could send the original messages as well.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Wednesday, July 25, 2012 at 12:49 AM, mete wrote:
>
> Hello folks,
>
> I am using flume-ng for cdh4 (1.10), and i am redirecting syslog output
> from a network device to flume-ng. My config is as follows:
> test1.channels.mem-chan-1.type = memory
> test1.channels.mem-chan-1.capacity = 100000
> test1.channels.mem-chan-1.transactionCapacity = 1000
>
> test1.sources.syslog-traffic.channels = mem-chan-1
> test1.sources.syslog-traffic.type = syslogudp
> test1.sources.syslog-traffic.port = 5140
> test1.sources.syslog-traffic.bind = test1
> test1.sources.syslog-traffic.eventSize = 10000
>
> test1.sinks.file-sink-1.channel = mem-chan-1
> test1.sinks.file-sink-1.type = file_roll
> test1.sinks.file-sink-1.sink.directory = /home/cloudera-user/tmp/
> test1.sinks.file-sink-1.rollInterval = 86400
>
> test1.channels = mem-chan-1
> test1.sources = syslog-traffic
> test1.sinks = file-sink-1
>
> i have a pretty straightforward config with one syslogudp source , a
> memory channel and a file sink.
>
> However, some of the messages i see on the file is like this:
>
> DEVICE: "some syslog content"@
> DEVICE: "some syslog content"@
> OUT^@
> FIN^@
> RST^@
> RST^@
> OUT^@
> FIN^@
> RST^@
> FIN^@
> FIN^@
> OUT^@
> RST^@
> RST^@
> RST^@
>
> As you can see, some lines are somehow trimmed and does not contain the
> entire message. When i redirect same device to syslog-ng there are no
> issues like this.
> I tried increasing the event size on the syslog source but that did not
> change anything at all.
> Any ideas on what might be the problem?
> Thanks in advance.
>
> Mete
>
>
>
Re: flume-ng & syslog source
Posted by Hari Shreedharan <hs...@cloudera.com>.
It would be helpful if you could send the original messages as well.
Thanks
Hari
--
Hari Shreedharan
On Wednesday, July 25, 2012 at 12:49 AM, mete wrote:
> Hello folks,
>
> I am using flume-ng for cdh4 (1.10), and i am redirecting syslog output from a network device to flume-ng. My config is as follows:
> test1.channels.mem-chan-1.type = memory
> test1.channels.mem-chan-1.capacity = 100000
> test1.channels.mem-chan-1.transactionCapacity = 1000
>
> test1.sources.syslog-traffic.channels = mem-chan-1
> test1.sources.syslog-traffic.type = syslogudp
> test1.sources.syslog-traffic.port = 5140
> test1.sources.syslog-traffic.bind = test1
> test1.sources.syslog-traffic.eventSize = 10000
>
> test1.sinks.file-sink-1.channel = mem-chan-1
> test1.sinks.file-sink-1.type = file_roll
> test1.sinks.file-sink-1.sink.directory = /home/cloudera-user/tmp/
> test1.sinks.file-sink-1.rollInterval = 86400
>
> test1.channels = mem-chan-1
> test1.sources = syslog-traffic
> test1.sinks = file-sink-1
>
>
> i have a pretty straightforward config with one syslogudp source , a memory channel and a file sink.
>
> However, some of the messages i see on the file is like this:
>
> DEVICE: "some syslog content"@
> DEVICE: "some syslog content"@
> OUT^@
> FIN^@
> RST^@
> RST^@
> OUT^@
> FIN^@
> RST^@
> FIN^@
> FIN^@
> OUT^@
> RST^@
> RST^@
> RST^@
>
>
> As you can see, some lines are somehow trimmed and does not contain the entire message. When i redirect same device to syslog-ng there are no issues like this.
> I tried increasing the event size on the syslog source but that did not change anything at all.
> Any ideas on what might be the problem?
> Thanks in advance.
>
> Mete