You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by 罗辉 <lu...@ifeng.com> on 2016/08/26 08:00:47 UTC
Flume sink exeption:org.apache.flume.EventDeliveryException: Failed
to send events
Hi there
I am new to flume, and trying to use flume to tail a logfile and push the messages via flume to sparkstreaming. However , I got an exception like below:
16/08/26 15:55:24 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: master60, port: 31002 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:89)
at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
at org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:211)
at org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:272)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:349)
... 3 more
Caused by: java.io.IOException: Error connecting to master60/10.0.10.60:31002
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
... 10 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:496)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:452)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:365)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Here is my agent config in the conf file
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = avro
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = master60
a1.sinks.k1.port = 31002
I don't know why my flume sink can not bind to the port 31002 which I specified.
Any idea will be appreciated.
San.Luo
Re: Flume sink exeption:org.apache.flume.EventDeliveryException:
Failed to send events
Posted by Denes Arvay <de...@cloudera.com>.
Hi,
Isn't it possible that your spark streaming application is not running when
flume tries to send events to it? That might be the reason of flume not
being able to connect to the port 31002.
According to the Spark streaming + Flume integration guide (
http://spark.apache.org/docs/latest/streaming-flume-integration.html) when
you use the push-based approach "the streaming application needs to be up,
with the receiver scheduled and listening on the chosen port, for Flume to
be able push data."
On the other hand you might want to consider using the more reliable
pull-based approach.
Regards,
Denes
On Mon, Aug 29, 2016 at 4:02 AM 罗辉 <lu...@ifeng.com> wrote:
> Hi Iain,
>
> telnet is ok, if I switch the sink to logger. Please check below log:
>
> source:
>
> [hadoop@master60 flume]$ telnet localhost 44444
>
> Trying ::1...
>
> telnet: connect to address ::1: Connection refused
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> abc
>
> OK
>
> abc
>
> OK
>
> gggd
>
> OK
>
>
>
>
>
> Sink:
>
> 16/08/29 09:57:16 INFO node.Application: Starting Sink k1
>
> 16/08/29 09:57:16 INFO node.Application: Starting Source r1
>
> 16/08/29 09:57:16 INFO source.NetcatSource: Source starting
>
> 16/08/29 09:57:16 INFO source.NetcatSource: Created
> serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
>
> 16/08/29 09:58:10 INFO sink.LoggerSink: Event: { headers:{} body: 61 62 63
> 0D abc. }
>
> 16/08/29 09:58:14 INFO sink.LoggerSink: Event: { headers:{} body: 61 62 63
> 0D abc. }
>
> 16/08/29 09:58:16 INFO sink.LoggerSink: Event: { headers:{} body: 67 67 67
> 64 0D gggd. }
>
>
>
> Seems it is ok when the sink is logger , but when I change the sink type
> from logger to avro. It failed with previous exception.
>
>
>
>
>
>
>
> *发件人:* iain wright [mailto:iainwrig@gmail.com]
> *发送时间:* 2016年8月27日 0:13
> *收件人:* user@flume.apache.org
> *主题:* Re: Flume sink exeption:org.apache.flume.EventDeliveryException:
> Failed to send events
>
>
>
> Can you telnet master60 31002 from the server the flume agent is running
> on?
>
>
> Sent from my iPhone
>
>
> On Aug 26, 2016, at 1:00 AM, 罗辉 <lu...@ifeng.com> wrote:
>
> 31002
>
>
答复: Flume sink exeption:org.apache.flume.EventDeliveryException: Failed to send events
Posted by 罗辉 <lu...@ifeng.com>.
Hi Iain,
telnet is ok, if I switch the sink to logger. Please check below log:
source:
[hadoop@master60 flume]$ telnet localhost 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
abc
OK
abc
OK
gggd
OK
Sink:
16/08/29 09:57:16 INFO node.Application: Starting Sink k1
16/08/29 09:57:16 INFO node.Application: Starting Source r1
16/08/29 09:57:16 INFO source.NetcatSource: Source starting
16/08/29 09:57:16 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
16/08/29 09:58:10 INFO sink.LoggerSink: Event: { headers:{} body: 61 62 63 0D abc. }
16/08/29 09:58:14 INFO sink.LoggerSink: Event: { headers:{} body: 61 62 63 0D abc. }
16/08/29 09:58:16 INFO sink.LoggerSink: Event: { headers:{} body: 67 67 67 64 0D gggd. }
Seems it is ok when the sink is logger , but when I change the sink type from logger to avro. It failed with previous exception.
发件人: iain wright [mailto:iainwrig@gmail.com]
发送时间: 2016年8月27日 0:13
收件人: user@flume.apache.org
主题: Re: Flume sink exeption:org.apache.flume.EventDeliveryException: Failed to send events
Can you telnet master60 31002 from the server the flume agent is running on?
Sent from my iPhone
On Aug 26, 2016, at 1:00 AM, 罗辉 <lu...@ifeng.com>> wrote:
31002
Re: Flume sink exeption:org.apache.flume.EventDeliveryException: Failed to send events
Posted by iain wright <ia...@gmail.com>.
Can you telnet master60 31002 from the server the flume agent is running on?
Sent from my iPhone
> On Aug 26, 2016, at 1:00 AM, 罗辉 <lu...@ifeng.com> wrote:
>
> 31002