You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "tycho_yang (JIRA)" <ji...@apache.org> on 2017/04/05 04:00:44 UTC
[jira] [Comment Edited] (FLUME-3021) flume Elasticsearch 5.0 not
support
[ https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956265#comment-15956265 ]
tycho_yang edited comment on FLUME-3021 at 4/5/17 3:59 AM:
-----------------------------------------------------------
HI Yonghao Zou
Have you tested flume Kafka channels with your sink to elasticsearch5.x,The next one I'm testing is wrong。Can determine the elasticsearch cluster is normal
2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:82)] Kafka version : 0.9.0.1
2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:83)] Kafka commitId : 23c69d62a0cabf06
2017-04-05 11:49:38,908 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.KafkaChannel.createConsumerAndRecords(KafkaChannel.java:296)] Created new consumer to connect to Kafka
2017-04-05 11:49:39,202 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.kafka.KafkaSource.doStart(KafkaSource.java:520)] Kafka source s1 started.
2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: s1: Successfully registered new MBean.
2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: s1 started
2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatCompletionHandler.handle(AbstractCoordinator.java:623)] Attempt to heart beat failed since the group is rebalancing, try to re-join group.
2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 0 revoked.
2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 1 revoked.
2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 0 assigned.
2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 1 assigned.
2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 0 assigned.
2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 1 assigned.
2017-04-05 11:49:42,769 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:222)] Failed to commit transaction. Transaction rolled back.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:167)
at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:370)
at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:166)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:179)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:206)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
2017-04-05 11:49:42,771 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
was (Author: ytc301):
HI Yonghao Zou
Have you tested flume Kafka channels with your sink to elasticsearch5.x,The next one I'm testing is wrong。
2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:82)] Kafka version : 0.9.0.1
2017-04-05 11:49:38,907 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.<init>(AppInfoParser.java:83)] Kafka commitId : 23c69d62a0cabf06
2017-04-05 11:49:38,908 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.KafkaChannel.createConsumerAndRecords(KafkaChannel.java:296)] Created new consumer to connect to Kafka
2017-04-05 11:49:39,202 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.kafka.KafkaSource.doStart(KafkaSource.java:520)] Kafka source s1 started.
2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: s1: Successfully registered new MBean.
2017-04-05 11:49:39,203 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: s1 started
2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatCompletionHandler.handle(AbstractCoordinator.java:623)] Attempt to heart beat failed since the group is rebalancing, try to re-join group.
2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 0 revoked.
2017-04-05 11:49:41,293 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsRevoked(KafkaSource.java:616)] topic www-web - partition 1 revoked.
2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 0 assigned.
2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 1 assigned.
2017-04-05 11:49:41,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.kafka.ChannelRebalanceListener.onPartitionsAssigned(KafkaChannel.java:751)] topic www-web-ch-sink - partition 0 assigned.
2017-04-05 11:49:41,298 (PollableSourceRunner-KafkaSource-s1) [INFO - org.apache.flume.source.kafka.SourceRebalanceListener.onPartitionsAssigned(KafkaSource.java:623)] topic www-web - partition 1 assigned.
2017-04-05 11:49:42,769 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:222)] Failed to commit transaction. Transaction rolled back.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:167)
at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:370)
at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:166)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:179)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:206)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
2017-04-05 11:49:42,771 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
Can determine the elasticsearch cluster is normal
> flume Elasticsearch 5.0 not support
> -----------------------------------
>
> Key: FLUME-3021
> URL: https://issues.apache.org/jira/browse/FLUME-3021
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.7.0
> Reporter: tycho_yang
> Assignee: Yonghao Zou
> Fix For: 1.8.0
>
> Attachments: FLUME-3021-0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)