You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by shadyxu <sh...@gmail.com> on 2014/11/17 10:47:15 UTC

How to separate json message into different columns when using ES sink?

Hi everyone,

I am now using Flume to collect log into ElasticSearch. And the logs are in
json format. However, when I check them in ES, it seems that Flume has put
the entire json log in the @message attribute. Is there any config to do or
need I do some coding to separate them into different columns as ES always
does?

BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
1.5.0.1.

Any clue shall be appreciated.

Re: How to separate json message into different columns when using ES sink?

Posted by shadyxu <sh...@gmail.com>.
The just released Flume 1.5.2 seems have not fix or even notice the
problems I mention above. Have anyone encounter the same problems or am I
doing something wrong?

2014-11-17 17:47 GMT+08:00 shadyxu <sh...@gmail.com>:

> Hi everyone,
>
> I am now using Flume to collect log into ElasticSearch. And the logs are
> in json format. However, when I check them in ES, it seems that Flume has
> put the entire json log in the @message attribute. Is there any config to
> do or need I do some coding to separate them into different columns as ES
> always does?
>
> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
> 1.5.0.1.
>
> Any clue shall be appreciated.
>
>
>

Re: How to separate json message into different columns when using ES sink?

Posted by shadyxu <sh...@gmail.com>.
Thanks. But before that, is there any related patch which I can refer to?
I've googled for a long time and found nothing useful.

2014-11-19 14:34 GMT+08:00 Hari Shreedharan <hs...@cloudera.com>:

> Flume 1.5.2 was a targeted release for specific security fixes, and did
> not pull in other patches. Please wait for 1.6 release which should be
> happening in a few weeks, I guess.
>
> Thanks,
> Hari
>
>
> On Mon, Nov 17, 2014 at 1:48 AM, shadyxu <sh...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I am now using Flume to collect log into ElasticSearch. And the logs are
>> in json format. However, when I check them in ES, it seems that Flume has
>> put the entire json log in the @message attribute. Is there any config to
>> do or need I do some coding to separate them into different columns as ES
>> always does?
>>
>> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
>> 1.5.0.1.
>>
>> Any clue shall be appreciated.
>>
>>
>>
>
>

Re: How to separate json message into different columns when using ES sink?

Posted by Hari Shreedharan <hs...@cloudera.com>.
Flume 1.5.2 was a targeted release for specific security fixes, and did not pull in other patches. Please wait for 1.6 release which should be happening in a few weeks, I guess.


Thanks,
Hari

On Mon, Nov 17, 2014 at 1:48 AM, shadyxu <sh...@gmail.com> wrote:

> Hi everyone,
> I am now using Flume to collect log into ElasticSearch. And the logs are in
> json format. However, when I check them in ES, it seems that Flume has put
> the entire json log in the @message attribute. Is there any config to do or
> need I do some coding to separate them into different columns as ES always
> does?
> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
> 1.5.0.1.
> Any clue shall be appreciated.

Re: How to separate json message into different columns when using ES sink?

Posted by Nickolay Kolev <le...@gmail.com>.
Hi Hari,

As I said I am not java programmer and I do not think this is quality code.
It is suited for our situation - we log JSON messages into ElasticSearch
If this code can be base for a solution that can be part of future releases.
What I have made is custom serializer and changes in the sink to check is
payload is valid and skip invalid events.
More information can be found here:
https://github.com/lem0na/flume/commit/60d753ebc86eddda579e3b871b0a8ab62a0f2794

best regards,
Nickolay Kolev

On Wed, Nov 19, 2014 at 11:57 PM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:

> If you submit your patch to Flume, we can hopefully commit it to the sink
> bundled with Flume
>
> Thanks,
> Hari
>
>
> On Wed, Nov 19, 2014 at 12:42 AM, Nickolay Kolev <le...@gmail.com>
> wrote:
>
>>    Hi,
>>
>> It looks like ES sink is oriented to handle text payload and not json and
>> there are problems with serialization.
>> We have the same problem and I have developed patch and custom version
>> based on sources (two months ago)
>> I am not java developer so probably it is not the best solution but it
>> works.
>> Now it is hosted on company's internal repo but if you have interest I
>> can upload it on GitHub or Bitbucket
>>
>> best regards,
>> Nickolay Kolev
>>
>>
>> On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <sh...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I am now using Flume to collect log into ElasticSearch. And the logs are
>>> in json format. However, when I check them in ES, it seems that Flume has
>>> put the entire json log in the @message attribute. Is there any config to
>>> do or need I do some coding to separate them into different columns as ES
>>> always does?
>>>
>>> BTW, I found that ttl seems not working in ElasticSearch. I'm using
>>> Flume 1.5.0.1.
>>>
>>> Any clue shall be appreciated.
>>>
>>>
>>>
>>
>>
>

Re: How to separate json message into different columns when using ES sink?

Posted by Hari Shreedharan <hs...@cloudera.com>.
If you submit your patch to Flume, we can hopefully commit it to the sink bundled with Flume


Thanks,
Hari

On Wed, Nov 19, 2014 at 12:42 AM, Nickolay Kolev <le...@gmail.com>
wrote:

> Hi,
> It looks like ES sink is oriented to handle text payload and not json and
> there are problems with serialization.
> We have the same problem and I have developed patch and custom version
> based on sources (two months ago)
> I am not java developer so probably it is not the best solution but it
> works.
> Now it is hosted on company's internal repo but if you have interest I can
> upload it on GitHub or Bitbucket
> best regards,
> Nickolay Kolev
> On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <sh...@gmail.com> wrote:
>> Hi everyone,
>>
>> I am now using Flume to collect log into ElasticSearch. And the logs are
>> in json format. However, when I check them in ES, it seems that Flume has
>> put the entire json log in the @message attribute. Is there any config to
>> do or need I do some coding to separate them into different columns as ES
>> always does?
>>
>> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
>> 1.5.0.1.
>>
>> Any clue shall be appreciated.
>>
>>
>>

Re: How to separate json message into different columns when using ES sink?

Posted by Nickolay Kolev <le...@gmail.com>.
Hi shadyxu,

This is the repo: https://github.com/lem0na/flume
My paches are in "es-flume" branch
As i said there must be a better solution but this works.
Hope that in near future oficial ES sink will support JSON payload too.

best regards,
Nickolay Kolev

On Wed, Nov 19, 2014 at 11:02 AM, shadyxu <sh...@gmail.com> wrote:

> Hi Nickolay,
>
> Thank you and please upload the repo on Github.
>
> 2014-11-19 16:41 GMT+08:00 Nickolay Kolev <le...@gmail.com>:
>
>> Hi,
>>
>> It looks like ES sink is oriented to handle text payload and not json and
>> there are problems with serialization.
>> We have the same problem and I have developed patch and custom version
>> based on sources (two months ago)
>> I am not java developer so probably it is not the best solution but it
>> works.
>> Now it is hosted on company's internal repo but if you have interest I
>> can upload it on GitHub or Bitbucket
>>
>> best regards,
>> Nickolay Kolev
>>
>>
>> On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <sh...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I am now using Flume to collect log into ElasticSearch. And the logs are
>>> in json format. However, when I check them in ES, it seems that Flume has
>>> put the entire json log in the @message attribute. Is there any config to
>>> do or need I do some coding to separate them into different columns as ES
>>> always does?
>>>
>>> BTW, I found that ttl seems not working in ElasticSearch. I'm using
>>> Flume 1.5.0.1.
>>>
>>> Any clue shall be appreciated.
>>>
>>>
>>>
>>
>>
>

Re: How to separate json message into different columns when using ES sink?

Posted by shadyxu <sh...@gmail.com>.
Hi Nickolay,

Thank you and please upload the repo on Github.

2014-11-19 16:41 GMT+08:00 Nickolay Kolev <le...@gmail.com>:

> Hi,
>
> It looks like ES sink is oriented to handle text payload and not json and
> there are problems with serialization.
> We have the same problem and I have developed patch and custom version
> based on sources (two months ago)
> I am not java developer so probably it is not the best solution but it
> works.
> Now it is hosted on company's internal repo but if you have interest I can
> upload it on GitHub or Bitbucket
>
> best regards,
> Nickolay Kolev
>
>
> On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <sh...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I am now using Flume to collect log into ElasticSearch. And the logs are
>> in json format. However, when I check them in ES, it seems that Flume has
>> put the entire json log in the @message attribute. Is there any config to
>> do or need I do some coding to separate them into different columns as ES
>> always does?
>>
>> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
>> 1.5.0.1.
>>
>> Any clue shall be appreciated.
>>
>>
>>
>
>

Re: How to separate json message into different columns when using ES sink?

Posted by Nickolay Kolev <le...@gmail.com>.
Hi,

It looks like ES sink is oriented to handle text payload and not json and
there are problems with serialization.
We have the same problem and I have developed patch and custom version
based on sources (two months ago)
I am not java developer so probably it is not the best solution but it
works.
Now it is hosted on company's internal repo but if you have interest I can
upload it on GitHub or Bitbucket

best regards,
Nickolay Kolev


On Mon, Nov 17, 2014 at 11:47 AM, shadyxu <sh...@gmail.com> wrote:

> Hi everyone,
>
> I am now using Flume to collect log into ElasticSearch. And the logs are
> in json format. However, when I check them in ES, it seems that Flume has
> put the entire json log in the @message attribute. Is there any config to
> do or need I do some coding to separate them into different columns as ES
> always does?
>
> BTW, I found that ttl seems not working in ElasticSearch. I'm using Flume
> 1.5.0.1.
>
> Any clue shall be appreciated.
>
>
>