You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Deepak Subhramanian <de...@gmail.com> on 2014/04/07 12:08:35 UTC
Re: Json over netcat source
Hi Simeon,
I am facing the same problem. Did you find any solution ?
Thanks, Deepak
On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici
<si...@yahoo.com>wrote:
> Hi guys
>
> I would like to send json data to a flume netcat source and then send it
> to an elasticsearch sink that uses the
>
> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>
> It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as *'*org.elasticsearch.common.xcontent.XContentBuilder'.
>
> Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
> works.
>
> Thanks a lot
>
>
--
Deepak Subhramanian
Re: Json over netcat source
Posted by Deepak Subhramanian <de...@gmail.com>.
Sorry. My mistake. It is loading JSON data properly after the temporary fix.
On Thu, May 8, 2014 at 6:24 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
> Hi Ashish,
>
> Thanks for the solution. I made the changes and I can see the JSON message
> now. There is a JIRA raised on the same issue.
>
> https://issues.apache.org/jira/browse/FLUME-2126
>
>
> From Hive when I load JSON data it automatically splits JSON fields to
> different columns. For some reason the ESSink doesnt load in the same way.
> I am not sure if I am setting the correct type. There is a parameter es.
> input.json I have to set to true in hive table . Is there any similar
> variable I have to set for ESSink
>
> Here is the raw data I am getting in Kibana.
>
> {
> "_index": "test-2014-05-08",
> "_type": "parsed_logs",
> "_id": "7qSBgRx-Q_GLaCDWARs_Cg",
> "_score": null,
> "_source": {
> "@message": "{\"action\":{\"id\":\"00001\"}}",
> "@timestamp": "2014-05-08T16:48:44.180Z",
> "@type": "application/json",
> "@fields": {
> "_attachment_mimetype": "application/json",
> "timestamp": "1399567724180",
> "_type": "application/json",
> "type": "application/json"
> }
> },
> "sort": [
> 1399567724180
> ]
> }
>
>
>
> On Sun, Apr 13, 2014 at 4:56 PM, Ashish <pa...@gmail.com> wrote:
>
>> little more on the issue
>>
>> builder.field(fieldName, tmp); calls the XContentBuilder API where class
>> type is determined and appropriate method is called. Since tmp, which is
>> instance of XContentBuilder, doesn't match any of the defined if conditions
>> it goes to final else where the tmp.toString() is called, and field(String,
>> String) method is called so we get object address in index.
>>
>> Replacing
>> builder.field(fieldName, tmp);
>> with
>> builder.field(fieldName, tmp.string());
>>
>> shall make things work, but I am not sure if this would be the best way
>> to use the API.
>>
>> Got the answer from ES user list :)
>>
>> http://elasticsearch-users.115913.n3.nabble.com/Issue-with-posting-json-data-to-elastic-search-via-Flume-td4054017.html
>>
>> Can ES experts comment on the best way forward?
>>
>>
>>
>> On Sun, Apr 13, 2014 at 8:10 PM, Ashish <pa...@gmail.com> wrote:
>>
>>> Have been able to reproduce the problem locally using the existing test
>>> cases inside ES Sink. The problem does exist.
>>>
>>> Did some initial investigation, the framework is able to detect the JSON
>>> content and tries to add it as complex field.
>>> timestamp is added only if present in header.
>>>
>>> In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil
>>>
>>> public static void addComplexField(XContentBuilder builder, String
>>> fieldName,
>>> XContentType contentType, byte[] data) throws IOException {
>>> XContentParser parser = null;
>>> try {
>>> XContentBuilder tmp = jsonBuilder();
>>> parser = XContentFactory.xContent(contentType).createParser(data);
>>> parser.nextToken();
>>> tmp.copyCurrentStructure(parser);
>>> builder.field(fieldName, tmp); <<<< This is where the we might
>>> have an issue (real action is happening inside this method
>>> call)
>>>
>>> Can someone familiar with this part look further into this? I shall
>>> debug further as soon as I have free cycles.
>>>
>>> thanks
>>> ashish
>>>
>>>
>>>
>>> On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
>>> deepak.subhramanian@gmail.com> wrote:
>>>
>>>> Thanks Simon. I am also struggling with no luck. I tried using the
>>>> latest flume elastic search sink jar build from 1.5SNAPSHOT ,but still no
>>>> luck. I will try to see if it is an issue with elastic search api . When I
>>>> loaded json using hive it loaded JSON properly. But we have to pass a
>>>> property es.input.json in hive. Is there a way to pass the same in Flume.
>>>>
>>>> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>>>>
>>>>
>>>>
>>>>
>>>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>>>> TBLPROPERTIES('es.resource' = '...',
>>>>
>>>>
>>>>
>>>>
>>>> 'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>>>>
>>>>
>>>
>>>
>>> --
>>> thanks
>>> ashish
>>>
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>
>
> --
> Deepak Subhramanian
>
--
Deepak Subhramanian
Re: Json over netcat source
Posted by Deepak Subhramanian <de...@gmail.com>.
Hi Ashish,
Thanks for the solution. I made the changes and I can see the JSON message
now. There is a JIRA raised on the same issue.
https://issues.apache.org/jira/browse/FLUME-2126
>From Hive when I load JSON data it automatically splits JSON fields to
different columns. For some reason the ESSink doesnt load in the same way.
I am not sure if I am setting the correct type. There is a parameter es.
input.json I have to set to true in hive table . Is there any similar
variable I have to set for ESSink
Here is the raw data I am getting in Kibana.
{
"_index": "test-2014-05-08",
"_type": "parsed_logs",
"_id": "7qSBgRx-Q_GLaCDWARs_Cg",
"_score": null,
"_source": {
"@message": "{\"action\":{\"id\":\"00001\"}}",
"@timestamp": "2014-05-08T16:48:44.180Z",
"@type": "application/json",
"@fields": {
"_attachment_mimetype": "application/json",
"timestamp": "1399567724180",
"_type": "application/json",
"type": "application/json"
}
},
"sort": [
1399567724180
]
}
On Sun, Apr 13, 2014 at 4:56 PM, Ashish <pa...@gmail.com> wrote:
> little more on the issue
>
> builder.field(fieldName, tmp); calls the XContentBuilder API where class
> type is determined and appropriate method is called. Since tmp, which is
> instance of XContentBuilder, doesn't match any of the defined if conditions
> it goes to final else where the tmp.toString() is called, and field(String,
> String) method is called so we get object address in index.
>
> Replacing
> builder.field(fieldName, tmp);
> with
> builder.field(fieldName, tmp.string());
>
> shall make things work, but I am not sure if this would be the best way to
> use the API.
>
> Got the answer from ES user list :)
>
> http://elasticsearch-users.115913.n3.nabble.com/Issue-with-posting-json-data-to-elastic-search-via-Flume-td4054017.html
>
> Can ES experts comment on the best way forward?
>
>
>
> On Sun, Apr 13, 2014 at 8:10 PM, Ashish <pa...@gmail.com> wrote:
>
>> Have been able to reproduce the problem locally using the existing test
>> cases inside ES Sink. The problem does exist.
>>
>> Did some initial investigation, the framework is able to detect the JSON
>> content and tries to add it as complex field.
>> timestamp is added only if present in header.
>>
>> In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil
>>
>> public static void addComplexField(XContentBuilder builder, String
>> fieldName,
>> XContentType contentType, byte[] data) throws IOException {
>> XContentParser parser = null;
>> try {
>> XContentBuilder tmp = jsonBuilder();
>> parser = XContentFactory.xContent(contentType).createParser(data);
>> parser.nextToken();
>> tmp.copyCurrentStructure(parser);
>> builder.field(fieldName, tmp); <<<< This is where the we might have
>> an issue (real action is happening inside this method
>> call)
>>
>> Can someone familiar with this part look further into this? I shall debug
>> further as soon as I have free cycles.
>>
>> thanks
>> ashish
>>
>>
>>
>> On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
>> deepak.subhramanian@gmail.com> wrote:
>>
>>> Thanks Simon. I am also struggling with no luck. I tried using the
>>> latest flume elastic search sink jar build from 1.5SNAPSHOT ,but still no
>>> luck. I will try to see if it is an issue with elastic search api . When I
>>> loaded json using hive it loaded JSON properly. But we have to pass a
>>> property es.input.json in hive. Is there a way to pass the same in Flume.
>>>
>>> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>>>
>>>
>>>
>>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>>> TBLPROPERTIES('es.resource' = '...',
>>>
>>>
>>>
>>> 'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>>>
>>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
--
Deepak Subhramanian
Re: Json over netcat source
Posted by Ashish <pa...@gmail.com>.
little more on the issue
builder.field(fieldName, tmp); calls the XContentBuilder API where class
type is determined and appropriate method is called. Since tmp, which is
instance of XContentBuilder, doesn't match any of the defined if conditions
it goes to final else where the tmp.toString() is called, and field(String,
String) method is called so we get object address in index.
Replacing
builder.field(fieldName, tmp);
with
builder.field(fieldName, tmp.string());
shall make things work, but I am not sure if this would be the best way to
use the API.
Got the answer from ES user list :)
http://elasticsearch-users.115913.n3.nabble.com/Issue-with-posting-json-data-to-elastic-search-via-Flume-td4054017.html
Can ES experts comment on the best way forward?
On Sun, Apr 13, 2014 at 8:10 PM, Ashish <pa...@gmail.com> wrote:
> Have been able to reproduce the problem locally using the existing test
> cases inside ES Sink. The problem does exist.
>
> Did some initial investigation, the framework is able to detect the JSON
> content and tries to add it as complex field.
> timestamp is added only if present in header.
>
> In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil
>
> public static void addComplexField(XContentBuilder builder, String
> fieldName,
> XContentType contentType, byte[] data) throws IOException {
> XContentParser parser = null;
> try {
> XContentBuilder tmp = jsonBuilder();
> parser = XContentFactory.xContent(contentType).createParser(data);
> parser.nextToken();
> tmp.copyCurrentStructure(parser);
> builder.field(fieldName, tmp); <<<< This is where the we might have
> an issue (real action is happening inside this method
> call)
>
> Can someone familiar with this part look further into this? I shall debug
> further as soon as I have free cycles.
>
> thanks
> ashish
>
>
>
> On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>> Thanks Simon. I am also struggling with no luck. I tried using the
>> latest flume elastic search sink jar build from 1.5SNAPSHOT ,but still no
>> luck. I will try to see if it is an issue with elastic search api . When I
>> loaded json using hive it loaded JSON properly. But we have to pass a
>> property es.input.json in hive. Is there a way to pass the same in Flume.
>>
>> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>>
>>
>> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
>> TBLPROPERTIES('es.resource' = '...',
>>
>>
>> 'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>>
>>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Json over netcat source
Posted by Ashish <pa...@gmail.com>.
Have been able to reproduce the problem locally using the existing test
cases inside ES Sink. The problem does exist.
Did some initial investigation, the framework is able to detect the JSON
content and tries to add it as complex field.
timestamp is added only if present in header.
In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil
public static void addComplexField(XContentBuilder builder, String
fieldName,
XContentType contentType, byte[] data) throws IOException {
XContentParser parser = null;
try {
XContentBuilder tmp = jsonBuilder();
parser = XContentFactory.xContent(contentType).createParser(data);
parser.nextToken();
tmp.copyCurrentStructure(parser);
builder.field(fieldName, tmp); <<<< This is where the we might have
an issue (real action is happening inside this method
call)
Can someone familiar with this part look further into this? I shall debug
further as soon as I have free cycles.
thanks
ashish
On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
> Thanks Simon. I am also struggling with no luck. I tried using the latest
> flume elastic search sink jar build from 1.5SNAPSHOT ,but still no luck. I
> will try to see if it is an issue with elastic search api . When I loaded
> json using hive it loaded JSON properly. But we have to pass a property
> es.input.json in hive. Is there a way to pass the same in Flume.
>
> CREATE EXTERNAL TABLE json (data STRING <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = '...',
>
> 'es.input.json` = 'yes' <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Json over netcat source
Posted by Deepak Subhramanian <de...@gmail.com>.
Thanks Simon. I am also struggling with no luck. I tried using the latest
flume elastic search sink jar build from 1.5SNAPSHOT ,but still no luck. I
will try to see if it is an issue with elastic search api . When I loaded
json using hive it loaded JSON properly. But we have to pass a property
es.input.json in hive. Is there a way to pass the same in Flume.
CREATE EXTERNAL TABLE json (data STRING
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = '...',
'es.input.json` = 'yes'
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
Re: Json over netcat source
Posted by Simeon Zaharici <si...@yahoo.com>.
Hello
I followed the instructions here https://flume.apache.org/FlumeUserGuide.html#elasticsearchsink and I am using a 0.90.7 jar on the flume server against an elastic search cluster running 0.90.7.
Also sending plain text through the netcat source works fine. The only problem is when the text that is sent is formatted as json
If I send the words plain text over the netcat source it ends up like this in elasticsearch
{"@message":"plain text\r","@timestamp":"2014-04-10T14:16:38.113Z","@fields":{"timestamp":"1397139398113"}}
If I send { “plain” : “text” }
{"@message":"org.elasticsearch.common.xcontent.XContentBuilder@4710ad01","@timestamp":"2014-04-10T14:20:00.871Z","@fields":{"timestamp":"1397139600871"}}
Also sending just {} over the netcat source results in the same error.
Thanks,
Simeon
On Thursday, April 10, 2014 2:48:49 AM, Mike Percy <mp...@apache.org> wrote:
Not sure either but make sure you're using a compatible version of ElasticSearch.
Sent from my iPhone
On Apr 9, 2014, at 9:43 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
Then I really don't know what the issue is. Someone more familiar with elastic search sink will need to look at it.
>
>
>Hari
>
>On Wednesday, April 9, 2014, Simeon Zaharici <si...@yahoo.com> wrote:
>
>Hello
>>
>>
>>This is to be able to easily integrate log centralization clients such as nxlog with flume
>>
>>
>>The problem is not related to the splitting, or the size, even a message like { "blah": "blah" } will result in the exception "org.elasticsearch.common.xcontent.XContentBuilder"
>>
>>
>>Simeon
>>On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>>
>>Why exactly do you want to send JSON over netcat? Why don't you simply use the Avro Source with Flume SDK? I am not entirely sure about the Elastic search sink. The reason your JSON is not working is likely because netcat is splitting the events per newline or when it reaches a fixed size.
>>
>>
>>
>>
>>Hari
>>
>>
>>
>>On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <si...@yahoo.com> wrote:
>>
>>Hello
>>>
>>>
>>>No, I did not find any solution to this
>>>
>>>
>>>Simeon
>>>On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <de...@gmail.com> wrote:
>>>
>>>Hi Simeon,
>>>I am facing the same problem. Did you find any solution ?
>>>
>>>
>>>Thanks, Deepak
>>>
>>>
>>>
>>>On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <si...@yahoo.com> wrote:
>>>
>>>Hi guys
>>>>
>>>>
>>>>I would like to send json data to a flume netcat source and then send it to an elasticsearch sink that uses the
>>>>org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>>>>
>>>>It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XContentBuilder'.
>>>>
>>>>Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
works.
>>>>
>>>>Thanks a lot
>>>>
>>>
>>>
>>>
>>>--
>>>Deepak Subhramanian
>>>
>>>
>>
>>
>>
Re: Json over netcat source
Posted by Mike Percy <mp...@apache.org>.
Not sure either but make sure you're using a compatible version of ElasticSearch.
Sent from my iPhone
> On Apr 9, 2014, at 9:43 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>
> Then I really don't know what the issue is. Someone more familiar with elastic search sink will need to look at it.
>
> Hari
>
>> On Wednesday, April 9, 2014, Simeon Zaharici <si...@yahoo.com> wrote:
>> Hello
>>
>> This is to be able to easily integrate log centralization clients such as nxlog with flume
>>
>> The problem is not related to the splitting, or the size, even a message like { "blah": "blah" } will result in the exception "org.elasticsearch.common.xcontent.XContentBuilder"
>>
>>
>> Simeon
>> On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>> Why exactly do you want to send JSON over netcat? Why don't you simply use the Avro Source with Flume SDK? I am not entirely sure about the Elastic search sink. The reason your JSON is not working is likely because netcat is splitting the events per newline or when it reaches a fixed size.
>>
>>
>> Hari
>>
>>
>> On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <si...@yahoo.com> wrote:
>> Hello
>>
>> No, I did not find any solution to this
>>
>> Simeon
>> On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <de...@gmail.com> wrote:
>> Hi Simeon,
>> I am facing the same problem. Did you find any solution ?
>>
>> Thanks, Deepak
>>
>>
>> On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <si...@yahoo.com> wrote:
>> Hi guys
>>
>> I would like to send json data to a flume netcat source and then send it to an elasticsearch sink that uses the
>> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>>
>>
>>
>>
>> It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XContentBuilder'.
>>
>>
>> Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
>> works.
>>
>> Thanks a lot
>>
>>
>>
>> --
>> Deepak Subhramanian
Re: Json over netcat source
Posted by Hari Shreedharan <hs...@cloudera.com>.
Then I really don't know what the issue is. Someone more familiar with
elastic search sink will need to look at it.
Hari
On Wednesday, April 9, 2014, Simeon Zaharici <si...@yahoo.com>
wrote:
> Hello
>
> This is to be able to easily integrate log centralization clients such as
> nxlog with flume
>
> The problem is not related to the splitting, or the size, even a message
> like { "blah": "blah" } will result in the exception "
> org.elasticsearch.common.xcontent.XContentBuilder"
>
> Simeon
> On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan <
> hshreedharan@cloudera.com<javascript:_e(%7B%7D,'cvml','hshreedharan@cloudera.com');>>
> wrote:
> Why exactly do you want to send JSON over netcat? Why don't you simply
> use the Avro Source with Flume SDK? I am not entirely sure about the
> Elastic search sink. The reason your JSON is not working is likely because
> netcat is splitting the events per newline or when it reaches a fixed size.
>
>
> Hari
>
>
> On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <simeon_zaharici@yahoo.com<javascript:_e(%7B%7D,'cvml','simeon_zaharici@yahoo.com');>
> > wrote:
>
> Hello
>
> No, I did not find any solution to this
>
> Simeon
> On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com<javascript:_e(%7B%7D,'cvml','deepak.subhramanian@gmail.com');>>
> wrote:
> Hi Simeon,
> I am facing the same problem. Did you find any solution ?
>
> Thanks, Deepak
>
>
> On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <simeon_zaharici@yahoo.com<javascript:_e(%7B%7D,'cvml','simeon_zaharici@yahoo.com');>
> > wrote:
>
> Hi guys
>
> I would like to send json data to a flume netcat source and then send it
> to an elasticsearch sink that uses the
>
> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>
>
> It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as *'*org.elasticsearch.common.xcontent.XContentBuilder'.
>
> Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
> works.
>
> Thanks a lot
>
>
>
>
> --
> Deepak Subhramanian
>
>
>
>
>
>
Re: Json over netcat source
Posted by Simeon Zaharici <si...@yahoo.com>.
Hello
This is to be able to easily integrate log centralization clients such as nxlog with flume
The problem is not related to the splitting, or the size, even a message like { "blah": "blah" } will result in the exception "org.elasticsearch.common.xcontent.XContentBuilder"
Simeon
On Wednesday, April 9, 2014 9:09:18 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
Why exactly do you want to send JSON over netcat? Why don't you simply use the Avro Source with Flume SDK? I am not entirely sure about the Elastic search sink. The reason your JSON is not working is likely because netcat is splitting the events per newline or when it reaches a fixed size.
Hari
On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici <si...@yahoo.com> wrote:
Hello
>
>
>No, I did not find any solution to this
>
>
>Simeon
>On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <de...@gmail.com> wrote:
>
>Hi Simeon,
>I am facing the same problem. Did you find any solution ?
>
>
>Thanks, Deepak
>
>
>
>On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <si...@yahoo.com> wrote:
>
>Hi guys
>>
>>
>>I would like to send json data to a flume netcat source and then send it to an elasticsearch sink that uses the
>>org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>>
>>It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XContentBuilder'.
>>
>>Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
works.
>>
>>Thanks a lot
>>
>
>
>
>--
>Deepak Subhramanian
>
>
Re: Json over netcat source
Posted by Hari Shreedharan <hs...@cloudera.com>.
Why exactly do you want to send JSON over netcat? Why don't you simply use
the Avro Source with Flume SDK? I am not entirely sure about the Elastic
search sink. The reason your JSON is not working is likely because netcat
is splitting the events per newline or when it reaches a fixed size.
Hari
On Wed, Apr 9, 2014 at 5:58 PM, Simeon Zaharici
<si...@yahoo.com>wrote:
> Hello
>
> No, I did not find any solution to this
>
> Simeon
> On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
> Hi Simeon,
> I am facing the same problem. Did you find any solution ?
>
> Thanks, Deepak
>
>
> On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <simeon_zaharici@yahoo.com
> > wrote:
>
> Hi guys
>
> I would like to send json data to a flume netcat source and then send it
> to an elasticsearch sink that uses the
>
> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>
> It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as *'*org.elasticsearch.common.xcontent.XContentBuilder'.
>
> Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
> works.
>
> Thanks a lot
>
>
>
>
> --
> Deepak Subhramanian
>
>
>
Re: Json over netcat source
Posted by Simeon Zaharici <si...@yahoo.com>.
Hello
No, I did not find any solution to this
Simeon
On Monday, April 7, 2014 6:09:21 AM, Deepak Subhramanian <de...@gmail.com> wrote:
Hi Simeon,
I am facing the same problem. Did you find any solution ?
Thanks, Deepak
On Wed, Mar 5, 2014 at 5:52 PM, Simeon Zaharici <si...@yahoo.com> wrote:
Hi guys
>
>
>I would like to send json data to a flume netcat source and then send it to an elasticsearch sink that uses the
>org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.
>
>It works to send plain text on the netcat source, however when I send json the event ends up in elasticsearch as 'org.elasticsearch.common.xcontent.XContentBuilder'.
>
>Which would be the easiest way to go about this ? I saw from reading docs that creating a morphline interceptor could be the way to go but I did not fully understand how that
works.
>
>Thanks a lot
>
--
Deepak Subhramanian