You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Deepak Subhramanian <de...@gmail.com> on 2014/04/07 18:08:54 UTC
Flume ElasticSearch Sink not indexing JSON data
I am posting JSON data to ElasticSearch using Flume . But for some reason
it is not loading the correct data in the message. Instead of JSON message
it is storing message data as
"org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
I tried using the DynamicSerializer which determines the content-type. But
no luck.
Here is the sample data posted on Elastic Search.
{
"_index": "test_flume-2014-04-07",
"_type": "logs",
"_id": "M9E-33RQTy2kA6QhW6mSUw",
"_score": null,
"_source": {
"@message": "org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2
",
"@timestamp": "2014-04-07T09:49:26.490Z",
"@fields": {
"timestamp": "1396864166490"
}
},
"sort": [
1396864166490,
1396864166490
]
}
org.elasticsearch.common.xcontent.XContentBuilder
--
Deepak Subhramanian
Re: Flume ElasticSearch Sink not indexing JSON data
Posted by Deepak Subhramanian <de...@gmail.com>.
Is there any additional serializers in 1.5SNAPSHOT which can resolve the
JSON issue. I noticed that with the 1.5 SNAPSHOT it was not storing
timestamp field in elasticsearch. Is the default serializer changed from
Logstash or is it a bug. I was getting timestamp in 1.4 version .
On Fri, Apr 11, 2014 at 12:56 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
>
> I tried using the latest flume elastic search sink jar build from
> 1.5SNAPSHOT ,but still no luck. I will try to see if it is an issue with
> elastic search api . When I loaded json data using hive it loaded JSON
> properly. But we have to pass a property es.input.json in hive. Is there a
> way to pass a property in Flume.There is another user also posted the same
> issue. Any inputs ?
>
>
> On Wed, Apr 9, 2014 at 2:18 PM, Ashish <pa...@gmail.com> wrote:
>
>> IMHO, checkout the trunk, build it and then just use the agent config
>> from old setup. Keep the old setup as is.
>>
>> thanks
>> ashish
>>
>>
>> On Wed, Apr 9, 2014 at 6:17 PM, Deepak Subhramanian <
>> deepak.subhramanian@gmail.com> wrote:
>>
>>> Thanks Otis. I will give it a try. Do I have to replace the flume jars
>>> or add the flume jar to the plugins directory to override the current flume
>>> version.
>>>
>>>
>>> On Wed, Apr 9, 2014 at 2:19 AM, Otis Gospodnetic <
>>> otis.gospodnetic@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The ES Sink has been modified quite a bit in the trunk. Would it make
>>>> sense to give that a try?
>>>>
>>>> Otis
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>>
>>>> On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
>>>> deepak.subhramanian@gmail.com> wrote:
>>>>
>>>>> I am posting JSON data to ElasticSearch using Flume . But for some
>>>>> reason it is not loading the correct data in the message. Instead of JSON
>>>>> message it is storing message data as
>>>>> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>>>>>
>>>>> I tried using the DynamicSerializer which determines the content-type.
>>>>> But no luck.
>>>>>
>>>>> Here is the sample data posted on Elastic Search.
>>>>>
>>>>> {
>>>>>
>>>>> "_index": "test_flume-2014-04-07",
>>>>>
>>>>> "_type": "logs",
>>>>>
>>>>> "_id": "M9E-33RQTy2kA6QhW6mSUw",
>>>>>
>>>>> "_score": null,
>>>>>
>>>>> "_source": {
>>>>>
>>>>> "@message": "
>>>>> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>>>>>
>>>>> "@timestamp": "2014-04-07T09:49:26.490Z",
>>>>>
>>>>> "@fields": {
>>>>>
>>>>> "timestamp": "1396864166490"
>>>>>
>>>>> }
>>>>>
>>>>> },
>>>>>
>>>>> "sort": [
>>>>>
>>>>> 1396864166490,
>>>>>
>>>>> 1396864166490
>>>>>
>>>>> ]
>>>>>
>>>>> }
>>>>>
>>>>> org.elasticsearch.common.xcontent.XContentBuilder
>>>>> --
>>>>> Deepak Subhramanian
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak Subhramanian
>>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>
>
> --
> Deepak Subhramanian
>
--
Deepak Subhramanian
Re: Flume ElasticSearch Sink not indexing JSON data
Posted by Deepak Subhramanian <de...@gmail.com>.
I tried using the latest flume elastic search sink jar build from
1.5SNAPSHOT ,but still no luck. I will try to see if it is an issue with
elastic search api . When I loaded json data using hive it loaded JSON
properly. But we have to pass a property es.input.json in hive. Is there a
way to pass a property in Flume.There is another user also posted the same
issue. Any inputs ?
On Wed, Apr 9, 2014 at 2:18 PM, Ashish <pa...@gmail.com> wrote:
> IMHO, checkout the trunk, build it and then just use the agent config from
> old setup. Keep the old setup as is.
>
> thanks
> ashish
>
>
> On Wed, Apr 9, 2014 at 6:17 PM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>> Thanks Otis. I will give it a try. Do I have to replace the flume jars or
>> add the flume jar to the plugins directory to override the current flume
>> version.
>>
>>
>> On Wed, Apr 9, 2014 at 2:19 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> The ES Sink has been modified quite a bit in the trunk. Would it make
>>> sense to give that a try?
>>>
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>> On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
>>> deepak.subhramanian@gmail.com> wrote:
>>>
>>>> I am posting JSON data to ElasticSearch using Flume . But for some
>>>> reason it is not loading the correct data in the message. Instead of JSON
>>>> message it is storing message data as
>>>> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>>>>
>>>> I tried using the DynamicSerializer which determines the content-type.
>>>> But no luck.
>>>>
>>>> Here is the sample data posted on Elastic Search.
>>>>
>>>> {
>>>>
>>>> "_index": "test_flume-2014-04-07",
>>>>
>>>> "_type": "logs",
>>>>
>>>> "_id": "M9E-33RQTy2kA6QhW6mSUw",
>>>>
>>>> "_score": null,
>>>>
>>>> "_source": {
>>>>
>>>> "@message": "
>>>> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>>>>
>>>> "@timestamp": "2014-04-07T09:49:26.490Z",
>>>>
>>>> "@fields": {
>>>>
>>>> "timestamp": "1396864166490"
>>>>
>>>> }
>>>>
>>>> },
>>>>
>>>> "sort": [
>>>>
>>>> 1396864166490,
>>>>
>>>> 1396864166490
>>>>
>>>> ]
>>>>
>>>> }
>>>>
>>>> org.elasticsearch.common.xcontent.XContentBuilder
>>>> --
>>>> Deepak Subhramanian
>>>>
>>>
>>>
>>
>>
>> --
>> Deepak Subhramanian
>>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
--
Deepak Subhramanian
Re: Flume ElasticSearch Sink not indexing JSON data
Posted by Ashish <pa...@gmail.com>.
IMHO, checkout the trunk, build it and then just use the agent config from
old setup. Keep the old setup as is.
thanks
ashish
On Wed, Apr 9, 2014 at 6:17 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
> Thanks Otis. I will give it a try. Do I have to replace the flume jars or
> add the flume jar to the plugins directory to override the current flume
> version.
>
>
> On Wed, Apr 9, 2014 at 2:19 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Hi,
>>
>> The ES Sink has been modified quite a bit in the trunk. Would it make
>> sense to give that a try?
>>
>> Otis
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
>> deepak.subhramanian@gmail.com> wrote:
>>
>>> I am posting JSON data to ElasticSearch using Flume . But for some
>>> reason it is not loading the correct data in the message. Instead of JSON
>>> message it is storing message data as
>>> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>>>
>>> I tried using the DynamicSerializer which determines the content-type.
>>> But no luck.
>>>
>>> Here is the sample data posted on Elastic Search.
>>>
>>> {
>>>
>>> "_index": "test_flume-2014-04-07",
>>>
>>> "_type": "logs",
>>>
>>> "_id": "M9E-33RQTy2kA6QhW6mSUw",
>>>
>>> "_score": null,
>>>
>>> "_source": {
>>>
>>> "@message": "
>>> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>>>
>>> "@timestamp": "2014-04-07T09:49:26.490Z",
>>>
>>> "@fields": {
>>>
>>> "timestamp": "1396864166490"
>>>
>>> }
>>>
>>> },
>>>
>>> "sort": [
>>>
>>> 1396864166490,
>>>
>>> 1396864166490
>>>
>>> ]
>>>
>>> }
>>>
>>> org.elasticsearch.common.xcontent.XContentBuilder
>>> --
>>> Deepak Subhramanian
>>>
>>
>>
>
>
> --
> Deepak Subhramanian
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Flume ElasticSearch Sink not indexing JSON data
Posted by Deepak Subhramanian <de...@gmail.com>.
Thanks Otis. I will give it a try. Do I have to replace the flume jars or
add the flume jar to the plugins directory to override the current flume
version.
On Wed, Apr 9, 2014 at 2:19 AM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:
> Hi,
>
> The ES Sink has been modified quite a bit in the trunk. Would it make
> sense to give that a try?
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>> I am posting JSON data to ElasticSearch using Flume . But for some reason
>> it is not loading the correct data in the message. Instead of JSON message
>> it is storing message data as
>> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>>
>> I tried using the DynamicSerializer which determines the content-type.
>> But no luck.
>>
>> Here is the sample data posted on Elastic Search.
>>
>> {
>>
>> "_index": "test_flume-2014-04-07",
>>
>> "_type": "logs",
>>
>> "_id": "M9E-33RQTy2kA6QhW6mSUw",
>>
>> "_score": null,
>>
>> "_source": {
>>
>> "@message": "
>> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>>
>> "@timestamp": "2014-04-07T09:49:26.490Z",
>>
>> "@fields": {
>>
>> "timestamp": "1396864166490"
>>
>> }
>>
>> },
>>
>> "sort": [
>>
>> 1396864166490,
>>
>> 1396864166490
>>
>> ]
>>
>> }
>>
>> org.elasticsearch.common.xcontent.XContentBuilder
>> --
>> Deepak Subhramanian
>>
>
>
--
Deepak Subhramanian
Re: Flume ElasticSearch Sink not indexing JSON data
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
The ES Sink has been modified quite a bit in the trunk. Would it make
sense to give that a try?
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
> I am posting JSON data to ElasticSearch using Flume . But for some reason
> it is not loading the correct data in the message. Instead of JSON message
> it is storing message data as
> "org.elasticsearch.common.xcontent.XContentBuilder@32f4122e"
>
> I tried using the DynamicSerializer which determines the content-type. But
> no luck.
>
> Here is the sample data posted on Elastic Search.
>
> {
>
> "_index": "test_flume-2014-04-07",
>
> "_type": "logs",
>
> "_id": "M9E-33RQTy2kA6QhW6mSUw",
>
> "_score": null,
>
> "_source": {
>
> "@message": "
> org.elasticsearch.common.xcontent.XContentBuilder@58bf76d2",
>
> "@timestamp": "2014-04-07T09:49:26.490Z",
>
> "@fields": {
>
> "timestamp": "1396864166490"
>
> }
>
> },
>
> "sort": [
>
> 1396864166490,
>
> 1396864166490
>
> ]
>
> }
>
> org.elasticsearch.common.xcontent.XContentBuilder
> --
> Deepak Subhramanian
>