You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Paul Chavez <pc...@verticalsearchworks.com> on 2012/10/30 19:25:19 UTC
Help determining exit code from flume avro-client
I am working on getting the flume avro-client functionality working on Windows, and am currently stuck on how to determine if the file was sent successfully.
I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to guide me and have built the 1.4.0 snapshot successfully. I can manually send files along to flume using the ''org.apache.flume.client.avro.AvroCLIClient" class.
The full command I am using is invoked via batch file with the following single line:
"C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroCLIClient -H 10.20.2.8 -p 6240 -F%1
My intention is to invoke from Powershell like so:
.\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log
This is working well and I am able to successfully send IIS and our app logs to flume and then onto HDFS without issues. Now I am trying to automate the process, and need to know if the operation failed for some reason. If I simply turn off the flume source I can trigger an error which is dumped to console like so (using flume.root.logger=DEBUG,console in the log4j file):
2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable t
o open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: 6240 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:
401)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipeli
neSink.java:370)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
Despite the ERROR message being logged to the console, java.exe still exits with code 0, indicating success. Is there a way to use the console appender to trigger java.exe to send a non-0 exit code if an error occurs so the rest of my automation tooling can detect the error? I can parse the output with Powershell and have it send a non-0 exit code itself, but looking for a more 'native' way to do this.
thanks,
Paul Chavez
Re: Help determining exit code from flume avro-client
Posted by Hari Shreedharan <hs...@cloudera.com>.
It'd be great if you could submit the patch upstream to Flume! :-)
--
Hari Shreedharan
On Tuesday, October 30, 2012 at 4:06 PM, Paul Chavez wrote:
> Thank you, I was unsure if that was desired behavior or not. I updated the class as a POC to add exit codes for each of those catch blocks and it works great.
>
> Thanks,
> Paul
>
>
> From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> Sent: Tuesday, October 30, 2012 11:47 AM
> To: user@flume.apache.org (mailto:user@flume.apache.org)
> Subject: Re: Help determining exit code from flume avro-client
>
> Hi Paul.
>
> Thanks for reporting this. It does look like Flume does not return non-zero error code even if there was an exception. I filed https://issues.apache.org/jira/browse/FLUME-1670 to address this.
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
>
> On Tuesday, October 30, 2012 at 11:25 AM, Paul Chavez wrote:
>
> > I am working on getting the flume avro-client functionality working on Windows, and am currently stuck on how to determine if the file was sent successfully.
> >
> > I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to guide me and have built the 1.4.0 snapshot successfully. I can manually send files along to flume using the ''org.apache.flume.client.avro.AvroCLIClient" class.
> >
> > The full command I am using is invoked via batch file with the following single line:
> > "C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroCLIClient -H 10.20.2.8 -p 6240 -F%1
> >
> > My intention is to invoke from Powershell like so:
> > .\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log
> >
> > This is working well and I am able to successfully send IIS and our app logs to flume and then onto HDFS without issues. Now I am trying to automate the process, and need to know if the operation failed for some reason. If I simply turn off the flume source I can trigger an error which is dumped to console like so (using flume.root.logger=DEBUG,console in the log4j file):
> >
> >
> > 2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable t
> > o open connection to Flume. Exception follows.
> > org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: 6240 }: RPC connection error
> > at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> > at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> > at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> > at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> > at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
> > at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> > Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240
> > at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> > at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> > at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> > at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> > ... 5 more
> > Caused by: java.net.ConnectException: Connection refused: no further information
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> > at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:
> > 401)
> > at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipeli
> > neSink.java:370)
> > at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> >
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > 2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
> >
> > Despite the ERROR message being logged to the console, java.exe still exits with code 0, indicating success. Is there a way to use the console appender to trigger java.exe to send a non-0 exit code if an error occurs so the rest of my automation tooling can detect the error? I can parse the output with Powershell and have it send a non-0 exit code itself, but looking for a more 'native' way to do this.
> >
> > thanks,
> > Paul Chavez
> >
> >
> >
>
>
RE: Help determining exit code from flume avro-client
Posted by Paul Chavez <pc...@verticalsearchworks.com>.
Thank you, I was unsure if that was desired behavior or not. I updated the class as a POC to add exit codes for each of those catch blocks and it works great.
Thanks,
Paul
________________________________
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Tuesday, October 30, 2012 11:47 AM
To: user@flume.apache.org
Subject: Re: Help determining exit code from flume avro-client
Hi Paul.
Thanks for reporting this. It does look like Flume does not return non-zero error code even if there was an exception. I filed https://issues.apache.org/jira/browse/FLUME-1670 to address this.
Thanks,
Hari
--
Hari Shreedharan
On Tuesday, October 30, 2012 at 11:25 AM, Paul Chavez wrote:
I am working on getting the flume avro-client functionality working on Windows, and am currently stuck on how to determine if the file was sent successfully.
I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to guide me and have built the 1.4.0 snapshot successfully. I can manually send files along to flume using the ''org.apache.flume.client.avro.AvroCLIClient" class.
The full command I am using is invoked via batch file with the following single line:
"C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroCLIClient -H 10.20.2.8 -p 6240 -F%1
My intention is to invoke from Powershell like so:
.\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log
This is working well and I am able to successfully send IIS and our app logs to flume and then onto HDFS without issues. Now I am trying to automate the process, and need to know if the operation failed for some reason. If I simply turn off the flume source I can trigger an error which is dumped to console like so (using flume.root.logger=DEBUG,console in the log4j file):
2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable t
o open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: 6240 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:
401)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipeli
neSink.java:370)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
Despite the ERROR message being logged to the console, java.exe still exits with code 0, indicating success. Is there a way to use the console appender to trigger java.exe to send a non-0 exit code if an error occurs so the rest of my automation tooling can detect the error? I can parse the output with Powershell and have it send a non-0 exit code itself, but looking for a more 'native' way to do this.
thanks,
Paul Chavez
Re: Help determining exit code from flume avro-client
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Paul.
Thanks for reporting this. It does look like Flume does not return non-zero error code even if there was an exception. I filed https://issues.apache.org/jira/browse/FLUME-1670 to address this.
Thanks,
Hari
--
Hari Shreedharan
On Tuesday, October 30, 2012 at 11:25 AM, Paul Chavez wrote:
> I am working on getting the flume avro-client functionality working on Windows, and am currently stuck on how to determine if the file was sent successfully.
>
> I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to guide me and have built the 1.4.0 snapshot successfully. I can manually send files along to flume using the ''org.apache.flume.client.avro.AvroCLIClient" class.
>
> The full command I am using is invoked via batch file with the following single line:
> "C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroCLIClient -H 10.20.2.8 -p 6240 -F%1
>
> My intention is to invoke from Powershell like so:
> .\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log
>
> This is working well and I am able to successfully send IIS and our app logs to flume and then onto HDFS without issues. Now I am trying to automate the process, and need to know if the operation failed for some reason. If I simply turn off the flume source I can trigger an error which is dumped to console like so (using flume.root.logger=DEBUG,console in the log4j file):
>
>
> 2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable t
> o open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: 6240 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:
> 401)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipeli
> neSink.java:370)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>
> Despite the ERROR message being logged to the console, java.exe still exits with code 0, indicating success. Is there a way to use the console appender to trigger java.exe to send a non-0 exit code if an error occurs so the rest of my automation tooling can detect the error? I can parse the output with Powershell and have it send a non-0 exit code itself, but looking for a more 'native' way to do this.
>
> thanks,
> Paul Chavez
>
>