You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jarek Jarcec Cecho (Created) (JIRA)" <ji...@apache.org> on 2011/11/25 08:36:40 UTC
[jira] [Created] (FLUME-861) AvroSource is failing on
ClosedChannelException
AvroSource is failing on ClosedChannelException
-----------------------------------------------
Key: FLUME-861
URL: https://issues.apache.org/jira/browse/FLUME-861
Project: Flume
Issue Type: Bug
Affects Versions: NG alpha 2
Reporter: Jarek Jarcec Cecho
Attachments: hddev01.log, hddev01.properties, hddev02.log, hddev02.properties
I've noticed that flume-ng is not reconnecting avro source/sink combination after restart of the sink part.
Steps to reproduce:
1) Run flume-ng on two nodes
2) Start sending data
3) Turn off sink instance (I used CTRL+C)
4) Turn it on again
5) Avro source part will fail on:
2011-11-25 02:22:23,720 (Thread-1) [WARN - org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:257)] Unexpected exception from downstream.
java.nio.channels.ClosedChannelException
(entire error message is present in hddev01.log)
I've used two centos boxes where first was acting as an agent, reading local log file using ExecSource. Second machine was acting as a collector, waiting for input events and storing them on HDFS. Both machines were connected using AVRO sink+source combination. You can find configuration files for both machines with their logs as well attached to this JIRA bug.
I've executed both flume-ng instances using following commands:
./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev01 > hddev01.log 2>&1
./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev02 > hddev02.log 2>&1
Input file was created using following small bash script (it was executed after flume-ng was successfully started):
for i in `seq -w 01 180`; do echo $i; echo Yoda-$i >> /var/log/jarcec; sleep 1s; done
Please note that I applied patch from FLUME-858 in order to get DataStream file type working for this test, however I was observing same behavior without it as well (e.g. using default sequence file)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (FLUME-861) AvroSource is failing on
ClosedChannelException
Posted by "E. Sammer (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer closed FLUME-861.
---------------------------
> AvroSource is failing on ClosedChannelException
> -----------------------------------------------
>
> Key: FLUME-861
> URL: https://issues.apache.org/jira/browse/FLUME-861
> Project: Flume
> Issue Type: Bug
> Affects Versions: NG alpha 2
> Reporter: Jarek Jarcec Cecho
> Assignee: E. Sammer
> Fix For: NG alpha 2
>
> Attachments: hddev01.log, hddev01.properties, hddev02.log, hddev02.properties
>
>
> I've noticed that flume-ng is not reconnecting avro source/sink combination after restart of the sink part.
> Steps to reproduce:
> 1) Run flume-ng on two nodes
> 2) Start sending data
> 3) Turn off sink instance (I used CTRL+C)
> 4) Turn it on again
> 5) Avro source part will fail on:
> 2011-11-25 02:22:23,720 (Thread-1) [WARN - org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:257)] Unexpected exception from downstream.
> java.nio.channels.ClosedChannelException
> (entire error message is present in hddev01.log)
> I've used two centos boxes where first was acting as an agent, reading local log file using ExecSource. Second machine was acting as a collector, waiting for input events and storing them on HDFS. Both machines were connected using AVRO sink+source combination. You can find configuration files for both machines with their logs as well attached to this JIRA bug.
> I've executed both flume-ng instances using following commands:
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev01 > hddev01.log 2>&1
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev02 > hddev02.log 2>&1
> Input file was created using following small bash script (it was executed after flume-ng was successfully started):
> for i in `seq -w 01 180`; do echo $i; echo Yoda-$i >> /var/log/jarcec; sleep 1s; done
> Please note that I applied patch from FLUME-858 in order to get DataStream file type working for this test, however I was observing same behavior without it as well (e.g. using default sequence file)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-861) AvroSource is failing on
ClosedChannelException
Posted by "E. Sammer (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer resolved FLUME-861.
-----------------------------
Resolution: Duplicate
Fix Version/s: NG alpha 2
Assignee: E. Sammer
Duplicate of FLUME-827. Jarek: If FLUME-827 doesn't solve the issue please feel free to reopen this. Thanks!
> AvroSource is failing on ClosedChannelException
> -----------------------------------------------
>
> Key: FLUME-861
> URL: https://issues.apache.org/jira/browse/FLUME-861
> Project: Flume
> Issue Type: Bug
> Affects Versions: NG alpha 2
> Reporter: Jarek Jarcec Cecho
> Assignee: E. Sammer
> Fix For: NG alpha 2
>
> Attachments: hddev01.log, hddev01.properties, hddev02.log, hddev02.properties
>
>
> I've noticed that flume-ng is not reconnecting avro source/sink combination after restart of the sink part.
> Steps to reproduce:
> 1) Run flume-ng on two nodes
> 2) Start sending data
> 3) Turn off sink instance (I used CTRL+C)
> 4) Turn it on again
> 5) Avro source part will fail on:
> 2011-11-25 02:22:23,720 (Thread-1) [WARN - org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:257)] Unexpected exception from downstream.
> java.nio.channels.ClosedChannelException
> (entire error message is present in hddev01.log)
> I've used two centos boxes where first was acting as an agent, reading local log file using ExecSource. Second machine was acting as a collector, waiting for input events and storing them on HDFS. Both machines were connected using AVRO sink+source combination. You can find configuration files for both machines with their logs as well attached to this JIRA bug.
> I've executed both flume-ng instances using following commands:
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev01 > hddev01.log 2>&1
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev02 > hddev02.log 2>&1
> Input file was created using following small bash script (it was executed after flume-ng was successfully started):
> for i in `seq -w 01 180`; do echo $i; echo Yoda-$i >> /var/log/jarcec; sleep 1s; done
> Please note that I applied patch from FLUME-858 in order to get DataStream file type working for this test, however I was observing same behavior without it as well (e.g. using default sequence file)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-861) AvroSource is failing on
ClosedChannelException
Posted by "Jarek Jarcec Cecho (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162324#comment-13162324 ]
Jarek Jarcec Cecho commented on FLUME-861:
------------------------------------------
Problem was indeed solved. Thank you very much sir.
> AvroSource is failing on ClosedChannelException
> -----------------------------------------------
>
> Key: FLUME-861
> URL: https://issues.apache.org/jira/browse/FLUME-861
> Project: Flume
> Issue Type: Bug
> Affects Versions: NG alpha 2
> Reporter: Jarek Jarcec Cecho
> Assignee: E. Sammer
> Fix For: NG alpha 2
>
> Attachments: hddev01.log, hddev01.properties, hddev02.log, hddev02.properties
>
>
> I've noticed that flume-ng is not reconnecting avro source/sink combination after restart of the sink part.
> Steps to reproduce:
> 1) Run flume-ng on two nodes
> 2) Start sending data
> 3) Turn off sink instance (I used CTRL+C)
> 4) Turn it on again
> 5) Avro source part will fail on:
> 2011-11-25 02:22:23,720 (Thread-1) [WARN - org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:257)] Unexpected exception from downstream.
> java.nio.channels.ClosedChannelException
> (entire error message is present in hddev01.log)
> I've used two centos boxes where first was acting as an agent, reading local log file using ExecSource. Second machine was acting as a collector, waiting for input events and storing them on HDFS. Both machines were connected using AVRO sink+source combination. You can find configuration files for both machines with their logs as well attached to this JIRA bug.
> I've executed both flume-ng instances using following commands:
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev01 > hddev01.log 2>&1
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev02 > hddev02.log 2>&1
> Input file was created using following small bash script (it was executed after flume-ng was successfully started):
> for i in `seq -w 01 180`; do echo $i; echo Yoda-$i >> /var/log/jarcec; sleep 1s; done
> Please note that I applied patch from FLUME-858 in order to get DataStream file type working for this test, however I was observing same behavior without it as well (e.g. using default sequence file)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-861) AvroSource is failing on
ClosedChannelException
Posted by "Jarek Jarcec Cecho (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated FLUME-861:
-------------------------------------
Attachment: hddev02.properties
hddev01.properties
hddev02.log
hddev01.log
> AvroSource is failing on ClosedChannelException
> -----------------------------------------------
>
> Key: FLUME-861
> URL: https://issues.apache.org/jira/browse/FLUME-861
> Project: Flume
> Issue Type: Bug
> Affects Versions: NG alpha 2
> Reporter: Jarek Jarcec Cecho
> Attachments: hddev01.log, hddev01.properties, hddev02.log, hddev02.properties
>
>
> I've noticed that flume-ng is not reconnecting avro source/sink combination after restart of the sink part.
> Steps to reproduce:
> 1) Run flume-ng on two nodes
> 2) Start sending data
> 3) Turn off sink instance (I used CTRL+C)
> 4) Turn it on again
> 5) Avro source part will fail on:
> 2011-11-25 02:22:23,720 (Thread-1) [WARN - org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:257)] Unexpected exception from downstream.
> java.nio.channels.ClosedChannelException
> (entire error message is present in hddev01.log)
> I've used two centos boxes where first was acting as an agent, reading local log file using ExecSource. Second machine was acting as a collector, waiting for input events and storing them on HDFS. Both machines were connected using AVRO sink+source combination. You can find configuration files for both machines with their logs as well attached to this JIRA bug.
> I've executed both flume-ng instances using following commands:
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev01 > hddev01.log 2>&1
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev02 > hddev02.log 2>&1
> Input file was created using following small bash script (it was executed after flume-ng was successfully started):
> for i in `seq -w 01 180`; do echo $i; echo Yoda-$i >> /var/log/jarcec; sleep 1s; done
> Please note that I applied patch from FLUME-858 in order to get DataStream file type working for this test, however I was observing same behavior without it as well (e.g. using default sequence file)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira