You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by shushuai zhu <ss...@yahoo.com> on 2013/06/13 05:47:20 UTC
How to use Flume Client perl API to send data to ElasticSearch?
Hi,
I am sending data in JSON format using Flume Client perl API to Flume Avro source then ElasticSearchSink to an ElasticSearch engine. I could send the data to ElasticSearch engine since I saw new indexes created in the engine, but the data are not formatted correctly or partially lost since they could not be viewed via Kibana.
An example of the data in JSON format is:
{"@timestamp":"2013-05-28T23:59:43.300Z","@source_host":"Source host name","@type":"flume-input","@fields":{"FIELD_PRIORITY":"DEBUG","FIELD_CLASS":"procedure.engine","FIELD_FILE":"/scratch/abc/abc.log","FIELD_MESSAGE":"The debug message","FIELD_TIME":"1369785558004"}}
Earlier, I used LogStash to pull JSON data from Redis to ElasticSearch, and the LogStash conf file looks like:
input {
redis {
host => "abc"
type => "redis-input"
data_type => "list"
key => "logstash"
format => "json"
}
}
output {
elasticsearch {
embedded => false
bind_host => "abc"
host => "abc"
port => 9300
}
}
The data were correctly input to ElasticSearch and could be viewed via Kibana.
The sample perl API call for FlumeNG shown in Flume Client doc is:
use Flume::Client;
my $ng_client = Flume::Client::Transceiver::Socket->new(host => 'host name', port => 41414);
my $ng_requestor = Flume::Client::Requestor::FlumeNG->new(client => $ng_client);
my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers => {}, body => "hello, this is sent from perl (using FlumeNG)"}]);
print "$response\n"; # response will be 'OK' on success
I tried putting the JSON data in "body" or "headers" or some other ways in the above example, but could not get the right results in ElasticSearch. Could someone tell me how to use the API to send the above example data to ElasticSearch?
Shushuai
Re: How to use Flume Client perl API to send data to ElasticSearch?
Posted by shushuai zhu <ss...@yahoo.com>.
Forgot to mention that the perl Flume client API I used is:
https://github.com/hendrik-schumacher/perl-flume
Has anyone used it? Especially for ElasticSearchSink? Thanks.
Shushuai
________________________________
From: shushuai zhu <ss...@yahoo.com>
To: "user@flume.apache.org" <us...@flume.apache.org>
Sent: Wednesday, June 12, 2013 11:47 PM
Subject: How to use Flume Client perl API to send data to ElasticSearch?
Hi,
I am sending data in JSON format using Flume Client perl API to Flume Avro source then ElasticSearchSink to an ElasticSearch engine. I could send the data to ElasticSearch engine since I saw new indexes created in the engine, but the data are not formatted correctly or partially lost since they could not be viewed via Kibana.
An example of the data in JSON format is:
{"@timestamp":"2013-05-28T23:59:43.300Z","@source_host":"Source host name","@type":"flume-input","@fields":{"FIELD_PRIORITY":"DEBUG","FIELD_CLASS":"procedure.engine","FIELD_FILE":"/scratch/abc/abc.log","FIELD_MESSAGE":"The debug message","FIELD_TIME":"1369785558004"}}
Earlier, I used LogStash to pull JSON data from Redis to ElasticSearch, and the LogStash conf file looks like:
input {
redis {
host => "abc"
type => "redis-input"
data_type => "list"
key => "logstash"
format => "json"
}
}
output {
elasticsearch {
embedded => false
bind_host => "abc"
host => "abc"
port => 9300
}
}
The data were correctly input to ElasticSearch and could be viewed via Kibana.
The sample perl API call for FlumeNG shown in Flume Client doc is:
use Flume::Client;
my $ng_client = Flume::Client::Transceiver::Socket->new(host => 'host name', port => 41414);
my $ng_requestor = Flume::Client::Requestor::FlumeNG->new(client => $ng_client);
my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers => {}, body => "hello, this is sent from perl (using FlumeNG)"}]);
print "$response\n"; # response will be 'OK' on success
I tried putting the JSON data in "body" or "headers" or some other ways in the above example, but could not get the right results in ElasticSearch. Could someone tell me how to use the API to send the above example data to ElasticSearch?
Shushuai