You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Chan Chor Pang <ch...@indetail.co.jp> on 2016/12/08 04:37:09 UTC
impalad create a huge amount of IPv6 connection
hi everyone,
one of the impalad in our cluster create a huge amount of IPv6 connection
even i restart the impalad, the connections will just come back in 30 min
any one know the reason for it to create so much IPv6 connection?
$ ps -ef | grep impalad
1002 26751 104961 0 11:25 pts/0 00:00:00 grep impalad
impala 112687 4366 8 Dec07 ? 01:31:08
/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/impala/sbin-retail/impalad
--flagfile=/var/run/cloudera-scm-agent/process/22177-impala-IMPALAD/impala-conf/impalad_flags
$ sudo lsof -p 112687 | grep TCP | wc -l
24165
$ sudo lsof -p 112687 | grep IPv6 | wc -l
23987
Re: impalad create a huge amount of IPv6 connection
Posted by Chan Chor Pang <ch...@indetail.co.jp>.
Thanks for confirming.
I do get quite a lot of ReopenClient message from client-cache.cc
and i believe thats the case.
I1208 16:07:19.947564 111389 client-cache.cc:80] ReopenClient():
re-creating client for had001:22000
I1208 16:07:19.997666 111389 client-cache.cc:80] ReopenClient():
re-creating client for had001:22000
I1208 16:07:20.032250 111389 client-cache.cc:80] ReopenClient():
re-creating client for had001:22000
I1208 16:07:20.069442 111389 client-cache.cc:80] ReopenClient():
re-creating client for had001:22000
btw the impalad version is 2.5 from CDH5.7.1
On 12/9/16 3:21 AM, Sailesh Mukil wrote:
> Yes, this problem has been fixed by IMPALA-3875. It's a relatively
> small patch, so you can easily apply it without conflicts if you
> wanted to.
>
> On Thu, Dec 8, 2016 at 10:06 AM, Tim Armstrong
> <tarmstrong@cloudera.com <ma...@cloudera.com>> wrote:
>
> Maybe Sailesh can confirm, but it does look like it's an instance
> of the connections-not-timing-out problem.
>
> On Wed, Dec 7, 2016 at 11:30 PM, Chan Chor Pang
> <chin-sh@indetail.co.jp <ma...@indetail.co.jp>> wrote:
>
> update:
>
> this problem is fixed by restarting all impalad
>
> if i was correct, the problem is cause by
> https://issues.cloudera.org/browse/IMPALA-2799
> <https://issues.cloudera.org/browse/IMPALA-2799>
>
> and all other host is trying to reconnect to the "problem
> host" endlessly
>
>
> On 12/8/16 2:09 PM, Chan Chor Pang wrote:
>>
>> some more info apparently, some host try to connect to the
>> problem host use Ipv4 but the problem host response as IPv6 $
>> sudo lsof -p 14864 | grep 60827 impalad 14864 impala 1826u
>> IPv6 787440313 0t0 TCP had003:snapenetio->had058:60827
>> (ESTABLISHED) $ sudo lsof -p 111823 | grep 60827 impalad
>> 111823 impala 442u IPv4 472142649 0t0 TCP
>> had058:60827->had001:snapenetio (ESTABLISHED)
>> On 12/8/16 1:37 PM, Chan Chor Pang wrote:
>>> hi everyone, one of the impalad in our cluster create a huge
>>> amount of IPv6 connection even i restart the impalad, the
>>> connections will just come back in 30 min any one know the
>>> reason for it to create so much IPv6 connection? $ ps -ef |
>>> grep impalad 1002 26751 104961 0 11:25 pts/0
>>> 00:00:00 grep impalad impala 112687 4366 8 Dec07
>>> ? 01:31:08
>>> /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/impala/sbin-retail/impalad
>>> --flagfile=/var/run/cloudera-scm-agent/process/22177-impala-IMPALAD/impala-conf/impalad_flags
>>> $ sudo lsof -p 112687 | grep TCP | wc -l 24165 $ sudo lsof
>>> -p 112687 | grep IPv6 | wc -l 23987
>
--
---*------------------------------------------------*---*---*---*---
\u682a\u5f0f\u4f1a\u793eINDETAIL
\u30cb\u30a2\u30b7\u30e7\u30a2\u7dcf\u5408\u30b5\u30fc\u30d3\u30b9\u4e8b\u696d\u672c\u90e8
\u30b2\u30fc\u30e0\u30b5\u30fc\u30d3\u30b9\u4e8b\u696d\u90e8
\u9673\u3000\u695a\u9d6c
E-mail :chin-sh@indetail.co.jp
URL : http://www.indetail.co.jp
\u3010\u672d\u5e4c\u672c\u793e\uff0fLABO\uff0fLABO2\u3011
\u3012060-0042
\u672d\u5e4c\u5e02\u4e2d\u592e\u533a\u5927\u901a\u897f9\u4e01\u76ee3\u756a\u573033
\u30ad\u30bf\u30b3\u30fc\u30bb\u30f3\u30bf\u30fc\u30d3\u30eb\u30c7\u30a3\u30f3\u30b0
\uff08\u672d\u5e4c\u672c\u793e\uff0fLABO2\uff1a2\u968e\u3001LABO\uff1a9\u968e\uff09
TEL\uff1a011-206-9235 FAX\uff1a011-206-9236
\u3010\u6771\u4eac\u652f\u5e97\u3011
\u3012108-0014
\u6771\u4eac\u90fd\u6e2f\u533a\u829d5\u4e01\u76ee29\u756a20\u53f7 \u30af\u30ed\u30b9\u30aa\u30d5\u30a3\u30b9\u4e09\u7530
TEL\uff1a03-6809-6502 FAX\uff1a03-6809-6504
\u3010\u540d\u53e4\u5c4b\u30b5\u30c6\u30e9\u30a4\u30c8\u3011
\u3012460-0002
\u611b\u77e5\u770c\u540d\u53e4\u5c4b\u5e02\u4e2d\u533a\u4e38\u306e\u51853\u4e01\u76ee17\u756a24\u53f7 NAYUTA BLD
TEL\uff1a052-971-0086
Re: impalad create a huge amount of IPv6 connection
Posted by Sailesh Mukil <sa...@cloudera.com>.
Yes, this problem has been fixed by IMPALA-3875. It's a relatively small
patch, so you can easily apply it without conflicts if you wanted to.
On Thu, Dec 8, 2016 at 10:06 AM, Tim Armstrong <ta...@cloudera.com>
wrote:
> Maybe Sailesh can confirm, but it does look like it's an instance of the
> connections-not-timing-out problem.
>
> On Wed, Dec 7, 2016 at 11:30 PM, Chan Chor Pang <ch...@indetail.co.jp>
> wrote:
>
>> update:
>>
>> this problem is fixed by restarting all impalad
>>
>> if i was correct, the problem is cause by
>> https://issues.cloudera.org/browse/IMPALA-2799
>> and all other host is trying to reconnect to the "problem host" endlessly
>>
>>
>> On 12/8/16 2:09 PM, Chan Chor Pang wrote:
>>
>> some more info
>>
>> apparently, some host try to connect to the problem host use Ipv4
>> but the problem host response as IPv6
>>
>> $ sudo lsof -p 14864 | grep 60827
>> impalad 14864 impala 1826u IPv6 787440313 0t0 TCP had003:snapenetio->had058:60827 (ESTABLISHED)
>>
>>
>> $ sudo lsof -p 111823 | grep 60827
>> impalad 111823 impala 442u IPv4 472142649 0t0 TCP had058:60827->had001:snapenetio (ESTABLISHED)
>>
>>
>> On 12/8/16 1:37 PM, Chan Chor Pang wrote:
>>
>> hi everyone,
>>
>> one of the impalad in our cluster create a huge amount of IPv6 connection
>> even i restart the impalad, the connections will just come back in 30 min
>>
>> any one know the reason for it to create so much IPv6 connection?
>>
>>
>> $ ps -ef | grep impalad
>> 1002 26751 104961 0 11:25 pts/0 00:00:00 grep impalad
>> impala 112687 4366 8 Dec07 ? 01:31:08
>> /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/impala/sbin-retail/impalad
>> --flagfile=/var/run/cloudera-scm-agent/process/22177-impala-
>> IMPALAD/impala-conf/impalad_flags
>> $ sudo lsof -p 112687 | grep TCP | wc -l
>> 24165
>> $ sudo lsof -p 112687 | grep IPv6 | wc -l
>> 23987
>>
>>
>>
>>
>
Re: impalad create a huge amount of IPv6 connection
Posted by Tim Armstrong <ta...@cloudera.com>.
Maybe Sailesh can confirm, but it does look like it's an instance of the
connections-not-timing-out problem.
On Wed, Dec 7, 2016 at 11:30 PM, Chan Chor Pang <ch...@indetail.co.jp>
wrote:
> update:
>
> this problem is fixed by restarting all impalad
>
> if i was correct, the problem is cause by
> https://issues.cloudera.org/browse/IMPALA-2799
> and all other host is trying to reconnect to the "problem host" endlessly
>
>
> On 12/8/16 2:09 PM, Chan Chor Pang wrote:
>
> some more info
>
> apparently, some host try to connect to the problem host use Ipv4
> but the problem host response as IPv6
>
> $ sudo lsof -p 14864 | grep 60827
> impalad 14864 impala 1826u IPv6 787440313 0t0 TCP had003:snapenetio->had058:60827 (ESTABLISHED)
>
>
> $ sudo lsof -p 111823 | grep 60827
> impalad 111823 impala 442u IPv4 472142649 0t0 TCP had058:60827->had001:snapenetio (ESTABLISHED)
>
>
> On 12/8/16 1:37 PM, Chan Chor Pang wrote:
>
> hi everyone,
>
> one of the impalad in our cluster create a huge amount of IPv6 connection
> even i restart the impalad, the connections will just come back in 30 min
>
> any one know the reason for it to create so much IPv6 connection?
>
>
> $ ps -ef | grep impalad
> 1002 26751 104961 0 11:25 pts/0 00:00:00 grep impalad
> impala 112687 4366 8 Dec07 ? 01:31:08
> /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/impala/sbin-retail/impalad
> --flagfile=/var/run/cloudera-scm-agent/process/22177-
> impala-IMPALAD/impala-conf/impalad_flags
> $ sudo lsof -p 112687 | grep TCP | wc -l
> 24165
> $ sudo lsof -p 112687 | grep IPv6 | wc -l
> 23987
>
>
>
>
Re: impalad create a huge amount of IPv6 connection
Posted by Chan Chor Pang <ch...@indetail.co.jp>.
update:
this problem is fixed by restarting all impalad
if i was correct, the problem is cause by
https://issues.cloudera.org/browse/IMPALA-2799
and all other host is trying to reconnect to the "problem host" endlessly
On 12/8/16 2:09 PM, Chan Chor Pang wrote:
>
> some more info apparently, some host try to connect to the problem
> host use Ipv4 but the problem host response as IPv6 $ sudo lsof -p
> 14864 | grep 60827 impalad 14864 impala 1826u IPv6 787440313 0t0 TCP
> had003:snapenetio->had058:60827 (ESTABLISHED) $ sudo lsof -p 111823 |
> grep 60827 impalad 111823 impala 442u IPv4 472142649 0t0 TCP
> had058:60827->had001:snapenetio (ESTABLISHED)
>
> On 12/8/16 1:37 PM, Chan Chor Pang wrote:
>> hi everyone,
>>
>> one of the impalad in our cluster create a huge amount of IPv6
>> connection
>> even i restart the impalad, the connections will just come back in 30
>> min
>>
>> any one know the reason for it to create so much IPv6 connection?
>>
>>
>> $ ps -ef | grep impalad
>> 1002 26751 104961 0 11:25 pts/0 00:00:00 grep impalad
>> impala 112687 4366 8 Dec07 ? 01:31:08
>> /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/impala/sbin-retail/impalad
>> --flagfile=/var/run/cloudera-scm-agent/process/22177-impala-IMPALAD/impala-conf/impalad_flags
>>
>> $ sudo lsof -p 112687 | grep TCP | wc -l
>> 24165
>> $ sudo lsof -p 112687 | grep IPv6 | wc -l
>> 23987
>>
>>
>
Re: impalad create a huge amount of IPv6 connection
Posted by Chan Chor Pang <ch...@indetail.co.jp>.
some more info apparently, some host try to connect to the problem host
use Ipv4 but the problem host response as IPv6 $ sudo lsof -p 14864 |
grep 60827 impalad 14864 impala 1826u IPv6 787440313 0t0 TCP
had003:snapenetio->had058:60827 (ESTABLISHED) $ sudo lsof -p 111823 |
grep 60827 impalad 111823 impala 442u IPv4 472142649 0t0 TCP
had058:60827->had001:snapenetio (ESTABLISHED)
On 12/8/16 1:37 PM, Chan Chor Pang wrote:
> hi everyone,
>
> one of the impalad in our cluster create a huge amount of IPv6 connection
> even i restart the impalad, the connections will just come back in 30 min
>
> any one know the reason for it to create so much IPv6 connection?
>
>
> $ ps -ef | grep impalad
> 1002 26751 104961 0 11:25 pts/0 00:00:00 grep impalad
> impala 112687 4366 8 Dec07 ? 01:31:08
> /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/impala/sbin-retail/impalad
> --flagfile=/var/run/cloudera-scm-agent/process/22177-impala-IMPALAD/impala-conf/impalad_flags
>
> $ sudo lsof -p 112687 | grep TCP | wc -l
> 24165
> $ sudo lsof -p 112687 | grep IPv6 | wc -l
> 23987
>
>
--
---*------------------------------------------------*---*---*---*---
\u682a\u5f0f\u4f1a\u793eINDETAIL
\u30cb\u30a2\u30b7\u30e7\u30a2\u7dcf\u5408\u30b5\u30fc\u30d3\u30b9\u4e8b\u696d\u672c\u90e8
\u30b2\u30fc\u30e0\u30b5\u30fc\u30d3\u30b9\u4e8b\u696d\u90e8
\u9673\u3000\u695a\u9d6c
E-mail :chin-sh@indetail.co.jp
URL : http://www.indetail.co.jp
\u3010\u672d\u5e4c\u672c\u793e\uff0fLABO\uff0fLABO2\u3011
\u3012060-0042
\u672d\u5e4c\u5e02\u4e2d\u592e\u533a\u5927\u901a\u897f9\u4e01\u76ee3\u756a\u573033
\u30ad\u30bf\u30b3\u30fc\u30bb\u30f3\u30bf\u30fc\u30d3\u30eb\u30c7\u30a3\u30f3\u30b0
\uff08\u672d\u5e4c\u672c\u793e\uff0fLABO2\uff1a2\u968e\u3001LABO\uff1a9\u968e\uff09
TEL\uff1a011-206-9235 FAX\uff1a011-206-9236
\u3010\u6771\u4eac\u652f\u5e97\u3011
\u3012108-0014
\u6771\u4eac\u90fd\u6e2f\u533a\u829d5\u4e01\u76ee29\u756a20\u53f7 \u30af\u30ed\u30b9\u30aa\u30d5\u30a3\u30b9\u4e09\u7530
TEL\uff1a03-6809-6502 FAX\uff1a03-6809-6504
\u3010\u540d\u53e4\u5c4b\u30b5\u30c6\u30e9\u30a4\u30c8\u3011
\u3012460-0002
\u611b\u77e5\u770c\u540d\u53e4\u5c4b\u5e02\u4e2d\u533a\u4e38\u306e\u51853\u4e01\u76ee17\u756a24\u53f7 NAYUTA BLD
TEL\uff1a052-971-0086