You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Mrityunjay Kumar (JIRA)" <ji...@apache.org> on 2015/12/17 00:31:46 UTC
[jira] [Commented] (FLUME-2688) the avro sink in sinkgroup will
never close the connection when process failed
[ https://issues.apache.org/jira/browse/FLUME-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061127#comment-15061127 ]
Mrityunjay Kumar commented on FLUME-2688:
-----------------------------------------
I would like to get this assigned to me. Thanks
> the avro sink in sinkgroup will never close the connection when process failed
> ------------------------------------------------------------------------------
>
> Key: FLUME-2688
> URL: https://issues.apache.org/jira/browse/FLUME-2688
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.5.2
> Environment: flume 1.5.2 two flume client and two flume ha(load_balance-->round_bin)
> Reporter: simonchi
> Priority: Blocker
>
> 169,177 two app server with flume agent to collect log and send them to remote ha flume by avro sink to connect with avro source.
> 84,88 two ha flume to receive the log from 169,177 by avro source to connect
> now when the time going on ,the connection between app and ha flume become more and more,it should be two connection in every ha flume ,ri ght ?
> but actually,it is not, as follows:
> [root@usercenter ~]# netstat -ano | grep 9520
> tcp 0 0 :::9520 :::* LISTEN off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:26371 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:43354 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:44055 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:18355 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:19968 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:21389 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:39635 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:56417 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:26416 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:63654 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:28258 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:29226 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:51033 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:21432 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:25367 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:16682 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:22288 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:23637 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:12397 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:40406 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:48131 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:39718 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:26650 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:32709 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:13135 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:28231 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:14285 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:19528 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:29502 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:46403 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:55529 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:32832 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:14920 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:30913 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:55971 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:46014 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:29924 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:60584 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:51096 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:14947 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:27936 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:33280 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:20794 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:16101 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:49969 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:45663 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:18501 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:63662 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:18487 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:34131 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:49777 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:26972 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:24546 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:48143 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:59868 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:26420 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:39200 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:21346 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.169:29591 ESTABLISHED off (0.00/0/0)
> tcp 0 0 ::ffff:172.30.129.84:9520 ::ffff:172.30.128.177:35161 ESTABLISHED off (0.00/0/0)
> so ,I think ,in the source code
> while (sinkIterator.hasNext()) {
> Sink sink = sinkIterator.next();
> try {
> status = sink.process();
> break;
> } catch (Exception ex) {
> selector.informSinkFailed(sink);
> LOGGER.warn("Sink failed to consume event. "
> + "Attempting next sink if available.", ex);
> }
> }
> in catch block,you should add a logic :
> if the sink is avro sink,you should close the old connection when it create a new connection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)