You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Arif,Mubaraka" <ar...@heb.com> on 2016/05/02 21:44:06 UTC
Flume - elasticsearch sink fails
Hello,
Trying to flume tweets from Tibco-JMS source into elastic search sink.
Running Elasticsearch on our edge nodes independent of Hadoop.
Running cloudera Hadoop 5.4.8
On running the error is :
May 2, 2:15:14.047 PM
INFO
org.apache.flume.instrumentation.MonitoredCounterGroup
Component type: SINK, name: k1 started
May 2, 2:15:14.050 PM
WARN
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
[arcapl0021707:9200]
May 2, 2:15:14.052 PM
ERROR
org.apache.flume.lifecycle.LifecycleSupervisor
Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Flume configuration for ElasticSearch is :
#confgiure elastic search sink
test.sinks.k1.type = elasticsearch
test.sinks.k1.hostNames = 172.17.43.207:9200
test.sinks.k1.indexName = TwitterComp_index
test.sinks.k1.indexType = TwitterComp_type
test.sinks.k1.clusterName = twittersearch
test.sinks.k1.batchSize = 500
test.sinks.k1.ttl = 5d
test.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
Any help is greatly appreciated.
Thanks,
Muby
Re: Flume - elasticsearch sink fails
Posted by Kartik Vashishta <ka...@gmail.com>.
I think this will NOT work, it is not supported on the most recent releases
of flume and ES
On Mon, May 2, 2016 at 4:27 PM, Arif,Mubaraka <ar...@heb.com> wrote:
> Hello Iain,
>
>
>
> They are in my plugins.d dir of flume. Do I also need to have them in
> flume lib folder ?
>
>
>
> Thanks,
>
> Muby
>
>
>
> *From:* iain wright [mailto:iainwrig@gmail.com]
> *Sent:* Monday, May 02, 2016 4:25 PM
> *To:* user@flume.apache.org
>
> *Subject:* Re: Flume - elasticsearch sink fails
>
>
>
> Hi Muby,
>
>
>
> Are the ES and lucene-core jars in your flume lib folder?
>
>
>
>
> --
>
> Iain Wright
>
>
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
> On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <ar...@heb.com>
> wrote:
>
> Downgraded to Elasticsearch 1.7.5 and still get the same error :
>
>
>
> RROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> *From:* iain wright [mailto:iainwrig@gmail.com]
> *Sent:* Monday, May 02, 2016 2:52 PM
> *To:* user@flume.apache.org
> *Cc:* Mabrito,Greg
> *Subject:* Re: Flume - elasticsearch sink fails
>
>
>
> Hi,
>
>
>
> These look relevant to your goals:
>
>
>
> https://github.com/elastic/elasticsearch/issues/14187
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=>
>
>
>
> https://github.com/lucidfrontier45/ElasticsearchSink2
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=>
>
>
>
> HTH,
>
>
> --
>
> Iain Wright
>
>
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
> On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <ar...@heb.com>
> wrote:
>
> Hello,
>
>
>
> Trying to flume tweets from Tibco-JMS source into elastic search sink.
>
> Running Elasticsearch on our edge nodes independent of Hadoop.
>
> Running cloudera Hadoop 5.4.8
>
>
>
> On running the error is :
>
>
>
> May 2, 2:15:14.047 PM
>
> INFO
>
> org.apache.flume.instrumentation.MonitoredCounterGroup
>
> Component type: SINK, name: k1 started
>
> May 2, 2:15:14.050 PM
>
> WARN
>
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
>
> [arcapl0021707:9200]
>
> May 2, 2:15:14.052 PM
>
> ERROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> Flume configuration for ElasticSearch is :
>
>
>
> #confgiure elastic search sink
>
> test.sinks.k1.type = elasticsearch
>
> test.sinks.k1.hostNames = 172.17.43.207:9200
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=>
>
> test.sinks.k1.indexName = TwitterComp_index
>
> test.sinks.k1.indexType = TwitterComp_type
>
> test.sinks.k1.clusterName = twittersearch
>
> test.sinks.k1.batchSize = 500
>
> test.sinks.k1.ttl = 5d
>
> test.sinks.k1.serializer =
> org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
>
>
>
> Any help is greatly appreciated.
>
>
>
> Thanks,
>
> *Muby*
>
>
>
>
>
RE: Flume - elasticsearch sink fails
Posted by "Arif,Mubaraka" <ar...@heb.com>.
Hello Iain,
They are in my plugins.d dir of flume. Do I also need to have them in flume lib folder ?
Thanks,
Muby
From: iain wright [mailto:iainwrig@gmail.com]
Sent: Monday, May 02, 2016 4:25 PM
To: user@flume.apache.org
Subject: Re: Flume - elasticsearch sink fails
Hi Muby,
Are the ES and lucene-core jars in your flume lib folder?
--
Iain Wright
This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <ar...@heb.com>> wrote:
Downgraded to Elasticsearch 1.7.5 and still get the same error :
RROR
org.apache.flume.lifecycle.LifecycleSupervisor
Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
From: iain wright [mailto:iainwrig@gmail.com<ma...@gmail.com>]
Sent: Monday, May 02, 2016 2:52 PM
To: user@flume.apache.org<ma...@flume.apache.org>
Cc: Mabrito,Greg
Subject: Re: Flume - elasticsearch sink fails
Hi,
These look relevant to your goals:
https://github.com/elastic/elasticsearch/issues/14187<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=>
https://github.com/lucidfrontier45/ElasticsearchSink2<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=>
HTH,
--
Iain Wright
This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <ar...@heb.com>> wrote:
Hello,
Trying to flume tweets from Tibco-JMS source into elastic search sink.
Running Elasticsearch on our edge nodes independent of Hadoop.
Running cloudera Hadoop 5.4.8
On running the error is :
May 2, 2:15:14.047 PM
INFO
org.apache.flume.instrumentation.MonitoredCounterGroup
Component type: SINK, name: k1 started
May 2, 2:15:14.050 PM
WARN
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
[arcapl0021707:9200]
May 2, 2:15:14.052 PM
ERROR
org.apache.flume.lifecycle.LifecycleSupervisor
Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Flume configuration for ElasticSearch is :
#confgiure elastic search sink
test.sinks.k1.type = elasticsearch
test.sinks.k1.hostNames = 172.17.43.207:9200<https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=>
test.sinks.k1.indexName = TwitterComp_index
test.sinks.k1.indexType = TwitterComp_type
test.sinks.k1.clusterName = twittersearch
test.sinks.k1.batchSize = 500
test.sinks.k1.ttl = 5d
test.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
Any help is greatly appreciated.
Thanks,
Muby
Re: Flume - elasticsearch sink fails
Posted by iain wright <ia...@gmail.com>.
Hi Muby,
Are the ES and lucene-core jars in your flume lib folder?
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <ar...@heb.com> wrote:
> Downgraded to Elasticsearch 1.7.5 and still get the same error :
>
>
>
> RROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> *From:* iain wright [mailto:iainwrig@gmail.com]
> *Sent:* Monday, May 02, 2016 2:52 PM
> *To:* user@flume.apache.org
> *Cc:* Mabrito,Greg
> *Subject:* Re: Flume - elasticsearch sink fails
>
>
>
> Hi,
>
>
>
> These look relevant to your goals:
>
>
>
> https://github.com/elastic/elasticsearch/issues/14187
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=>
>
>
>
> https://github.com/lucidfrontier45/ElasticsearchSink2
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=>
>
>
>
> HTH,
>
>
> --
>
> Iain Wright
>
>
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
> On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <ar...@heb.com>
> wrote:
>
> Hello,
>
>
>
> Trying to flume tweets from Tibco-JMS source into elastic search sink.
>
> Running Elasticsearch on our edge nodes independent of Hadoop.
>
> Running cloudera Hadoop 5.4.8
>
>
>
> On running the error is :
>
>
>
> May 2, 2:15:14.047 PM
>
> INFO
>
> org.apache.flume.instrumentation.MonitoredCounterGroup
>
> Component type: SINK, name: k1 started
>
> May 2, 2:15:14.050 PM
>
> WARN
>
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
>
> [arcapl0021707:9200]
>
> May 2, 2:15:14.052 PM
>
> ERROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> Flume configuration for ElasticSearch is :
>
>
>
> #confgiure elastic search sink
>
> test.sinks.k1.type = elasticsearch
>
> test.sinks.k1.hostNames = 172.17.43.207:9200
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=>
>
> test.sinks.k1.indexName = TwitterComp_index
>
> test.sinks.k1.indexType = TwitterComp_type
>
> test.sinks.k1.clusterName = twittersearch
>
> test.sinks.k1.batchSize = 500
>
> test.sinks.k1.ttl = 5d
>
> test.sinks.k1.serializer =
> org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
>
>
>
> Any help is greatly appreciated.
>
>
>
> Thanks,
>
> *Muby*
>
>
>
RE: Flume - elasticsearch sink fails
Posted by "Arif,Mubaraka" <ar...@heb.com>.
Downgraded to Elasticsearch 1.7.5 and still get the same error :
RROR
org.apache.flume.lifecycle.LifecycleSupervisor
Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
From: iain wright [mailto:iainwrig@gmail.com]
Sent: Monday, May 02, 2016 2:52 PM
To: user@flume.apache.org
Cc: Mabrito,Greg
Subject: Re: Flume - elasticsearch sink fails
Hi,
These look relevant to your goals:
https://github.com/elastic/elasticsearch/issues/14187<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=>
https://github.com/lucidfrontier45/ElasticsearchSink2<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=>
HTH,
--
Iain Wright
This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <ar...@heb.com>> wrote:
Hello,
Trying to flume tweets from Tibco-JMS source into elastic search sink.
Running Elasticsearch on our edge nodes independent of Hadoop.
Running cloudera Hadoop 5.4.8
On running the error is :
May 2, 2:15:14.047 PM
INFO
org.apache.flume.instrumentation.MonitoredCounterGroup
Component type: SINK, name: k1 started
May 2, 2:15:14.050 PM
WARN
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
[arcapl0021707:9200]
May 2, 2:15:14.052 PM
ERROR
org.apache.flume.lifecycle.LifecycleSupervisor
Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Flume configuration for ElasticSearch is :
#confgiure elastic search sink
test.sinks.k1.type = elasticsearch
test.sinks.k1.hostNames = 172.17.43.207:9200<https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=>
test.sinks.k1.indexName = TwitterComp_index
test.sinks.k1.indexType = TwitterComp_type
test.sinks.k1.clusterName = twittersearch
test.sinks.k1.batchSize = 500
test.sinks.k1.ttl = 5d
test.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
Any help is greatly appreciated.
Thanks,
Muby
Re: Flume - elasticsearch sink fails
Posted by iain wright <ia...@gmail.com>.
Hi,
These look relevant to your goals:
https://github.com/elastic/elasticsearch/issues/14187
https://github.com/lucidfrontier45/ElasticsearchSink2
HTH,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <ar...@heb.com>
wrote:
> Hello,
>
>
>
> Trying to flume tweets from Tibco-JMS source into elastic search sink.
>
> Running Elasticsearch on our edge nodes independent of Hadoop.
>
> Running cloudera Hadoop 5.4.8
>
>
>
> On running the error is :
>
>
>
> May 2, 2:15:14.047 PM
>
> INFO
>
> org.apache.flume.instrumentation.MonitoredCounterGroup
>
> Component type: SINK, name: k1 started
>
> May 2, 2:15:14.050 PM
>
> WARN
>
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient
>
> [arcapl0021707:9200]
>
> May 2, 2:15:14.052 PM
>
> ERROR
>
> org.apache.flume.lifecycle.LifecycleSupervisor
>
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{
> name:null counters:{} } } - Exception follows.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77)
>
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
>
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> Flume configuration for ElasticSearch is :
>
>
>
> #confgiure elastic search sink
>
> test.sinks.k1.type = elasticsearch
>
> test.sinks.k1.hostNames = 172.17.43.207:9200
>
> test.sinks.k1.indexName = TwitterComp_index
>
> test.sinks.k1.indexType = TwitterComp_type
>
> test.sinks.k1.clusterName = twittersearch
>
> test.sinks.k1.batchSize = 500
>
> test.sinks.k1.ttl = 5d
>
> test.sinks.k1.serializer =
> org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
>
>
>
> Any help is greatly appreciated.
>
>
>
> Thanks,
>
> *Muby*
>