You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "liu jing (Jira)" <ji...@apache.org> on 2022/03/07 08:26:00 UTC
[jira] [Created] (KUDU-3358) kudu tables fail to insert and scan when k8s network changes
liu jing created KUDU-3358:
------------------------------
Summary: kudu tables fail to insert and scan when k8s network changes
Key: KUDU-3358
URL: https://issues.apache.org/jira/browse/KUDU-3358
Project: Kudu
Issue Type: Bug
Affects Versions: 1.10.0
Reporter: liu jing
h3. Description
When I use the k8s to manage the kudu's network, there is a problem that if the k8s restart or any other way to change the kudu pod's ip, then kudu's tables will fail to insert or scan.
h3. Make a reappear
There is a way to trigger the problem, using the impala to make a test.
1. First, the original k8s pod network like this figure1:
{panel:title=figure1}
service-kudu-test01-entry ClusterIP 10.98.78.224 <none> 8051/TCP,8050/TCP,7051/TCP,7050/TCP 2d22h service-kudu-test01-master-0 ClusterIP 10.109.78.49 <none> 7051/TCP,8051/TCP,20051/TCP 2d22h service-kudu-test01-master-1 ClusterIP 10.98.28.69 <none> 7051/TCP,8051/TCP,20051/TCP 2d22h service-kudu-test01-master-2 ClusterIP 10.105.180.113 <none> 7051/TCP,8051/TCP,20051/TCP 2d22h service-kudu-test01-{color:#FF0000}tserver-0 ClusterIP 10.106.224.20 <none> 7050/TCP,8050/TCP,20050/TCP 2d22h{color} {color:#FF0000}service-kudu-test01-tserver-1 ClusterIP 10.110.69.131 <none> 7050/TCP,8050/TCP,20050/TCP 2d22h{color} service-kudu-test01-tserver-2 ClusterIP 10.108.30.59 <none> 7050/TCP,8050/TCP,20050/TCP 2d22h
{panel}
2. Second, using impala to create a table named *testTable.*
3. Then, restart the pod service, using the command:
{code:java}
kubectl delete --force -f ${dirname}/xx.yaml
kubectl apply --force -f ${dirname}/xx.yaml{code}
This will lead the kudu pod service to another new network, like this:
{panel:title=figure2}
service-kudu-test01-entry ClusterIP 10.108.85.55 <none> 8051/TCP,8050/TCP,7051/TCP,7050/TCP 2m22s
service-kudu-test01-master-0 ClusterIP 10.96.245.192 <none> 7051/TCP,8051/TCP,20051/TCP 2m22s
service-kudu-test01-master-1 ClusterIP 10.105.96.68 <none> 7051/TCP,8051/TCP,20051/TCP 2m22s
service-kudu-test01-master-2 ClusterIP 10.103.221.65 <none> 7051/TCP,8051/TCP,20051/TCP 2m22s
{color:#FF0000}service-kudu-test01-tserver-0 ClusterIP 10.101.128.27 <none> 7050/TCP,8050/TCP,20050/TCP 2m22s{color}
{color:#FF0000}service-kudu-test01-tserver-1 ClusterIP 10.111.9.225 <none> 7050/TCP,8050/TCP,20050/TCP 2m22s{color}
service-kudu-test01-tserver-2 ClusterIP 10.104.26.31 <none> 7050/TCP,8050/TCP,20050/TCP 2m22s
{panel}
4. Then, using the impala to scan the table {*}testTable{*}, like this:
{code:java}
select * from testTable
{code}
then, the impala client return a error, like this:
{code:java}
[service-impala-test01-server-0:21000] default> select * from testTable;
Query: select * from testTable
Query submitted at: 2022-03-07 15:13:04 (Coordinator: http://service-impala-test01-server-0:25000)
Query progress can be monitored at: http://service-impala-test01-server-0:25000/query_plan?query_id=c84e8a34795ca311:953d6fd800000000
ERROR: Unable to open scanner for node with id '0' for Kudu table 'impala::default.testTable': Timed out: exceeded configured scan timeout of 180.000s: after 3 scan attempts: Client connection negotiation failed: client connection to 10.110.69.131:7050: Timeout exceeded waiting to connect: Network error: Client connection negotiation failed: client connection to 10.106.224.20:7050: connect: Connection refused (error 111) {code}
From this error log, we can find that kudu master return an old tserver ip to impala client(we can use *figure1* to check the ip) . But, this ip is not available, so impala fail to make a scan.
5. Depending on the new network, using the impala to create a new table {*}testTable2{*}. It will succeed. But, if we use impala to make a insert or select for the {*}testTable2{*}, it will return the same error log, like this:
{code:java}
ERROR: Unable to open scanner for node with id '0' for Kudu table 'impala::default.testTable2': Timed out: exceeded configured scan timeout of 180.000s: after 3 scan attempts: Client connection negotiation failed: client connection to 10.110.69.131:7050: Timeout exceeded waiting to connect: Network error: Client connection negotiation failed: client connection to 10.106.224.20:7050: connect: Connection refused (error 111) {code}
This indicates that the kudu master still uses the old network to manage the new table.
h3. To avoid the problem
If I use the local machine's network for kudu, the problem will not happen
--
This message was sent by Atlassian Jira
(v8.20.1#820001)