Handling multi-line messages?

[Scott Chapman <sc...@woofplanet.com>]

So, avoiding a bit of a long explanation on why I'm doing it this way...

But essentially, I am trying to put multi-line messages into kafka and then
parse them in logstash.

What I think I am seeing in kafka (using console-consumer) is this:
 "line 1 \nline 2 \nline 3\n"

Then when I get it into logstash I am seeing it as:
   {
"message" => "line 1 \\nline 2 \\nline \n",
"@version" => "1",
"@timestamp" => "2015-02-09T13:55:36.566Z",
  }

My question is, is this what I should expect? I think I can probably figure
out take the single line and break it apart in logstash. But do I need to?

Any thoughts?

Re: Handling multi-line messages?

[Scott Chapman <sc...@woofplanet.com>]

Yea, I think I figured it out. Didn't realize the person doing the test
created the message using the console-consumer, so I think the newline was
escaped.

On Mon Feb 09 2015 at 11:59:57 AM Gwen Shapira <gs...@cloudera.com>
wrote:

> Since the console-consumer seems to display strings correctly, it sounds
> like an issue with LogStash parser. Perhaps you'll have better luck asking
> on LogStash mailing list?
>
> Kafka just stores the bytes you put in and gives the same bytes out when
> you read messages. There's no parsing or encoding done in Kafka itself
> (other than the encoder/decoder you use in producer / consumer)
>
> Gwen
>
> On Mon, Feb 9, 2015 at 6:23 AM, Scott Chapman <sc...@woofplanet.com>
> wrote:
>
> > So, avoiding a bit of a long explanation on why I'm doing it this way...
> >
> > But essentially, I am trying to put multi-line messages into kafka and
> then
> > parse them in logstash.
> >
> > What I think I am seeing in kafka (using console-consumer) is this:
> >  "line 1 \nline 2 \nline 3\n"
> >
> > Then when I get it into logstash I am seeing it as:
> >    {
> > "message" => "line 1 \\nline 2 \\nline \n",
> > "@version" => "1",
> > "@timestamp" => "2015-02-09T13:55:36.566Z",
> >   }
> >
> > My question is, is this what I should expect? I think I can probably
> figure
> > out take the single line and break it apart in logstash. But do I need
> to?
> >
> > Any thoughts?
> >
>

Re: Handling multi-line messages?

[Gwen Shapira <gs...@cloudera.com>]

Since the console-consumer seems to display strings correctly, it sounds
like an issue with LogStash parser. Perhaps you'll have better luck asking
on LogStash mailing list?

Kafka just stores the bytes you put in and gives the same bytes out when
you read messages. There's no parsing or encoding done in Kafka itself
(other than the encoder/decoder you use in producer / consumer)

Gwen

On Mon, Feb 9, 2015 at 6:23 AM, Scott Chapman <sc...@woofplanet.com> wrote:

> So, avoiding a bit of a long explanation on why I'm doing it this way...
>
> But essentially, I am trying to put multi-line messages into kafka and then
> parse them in logstash.
>
> What I think I am seeing in kafka (using console-consumer) is this:
>  "line 1 \nline 2 \nline 3\n"
>
> Then when I get it into logstash I am seeing it as:
>    {
> "message" => "line 1 \\nline 2 \\nline \n",
> "@version" => "1",
> "@timestamp" => "2015-02-09T13:55:36.566Z",
>   }
>
> My question is, is this what I should expect? I think I can probably figure
> out take the single line and break it apart in logstash. But do I need to?
>
> Any thoughts?
>