You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by vbm <bm...@gmail.com> on 2018/06/20 02:28:04 UTC
Ignite on RBAC enabled K8s cluster
Hi,
I am trying to bring up Ignite on RBAC enabled K8s cluster. As part of
discovery, when ignite nodes are trying to discover one another, I am seeing
below error:
[02:16:08,755][SEVERE][tcp-disco-ip-finder-cleaner-#4][TcpDiscoverySpi]
Failed to clean IP finder up.
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite
pods IP addresses.
at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1810)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1921)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1896)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL:
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/ignite
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)
I see there is a ticket on it, but it uses a cluster-admin role. Not sure
why we need cluster-admin role.
I tried the steps mentioned in the ticket, but still I am getting the same
error.
I was not able to find any documentation on the same. Has anyone tried
bringing up Ignite on RBAC enabled K8s cluster ?
Regards,
Vishwas
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite on RBAC enabled K8s cluster
Posted by vbm <bm...@gmail.com>.
Hi Roman,
Thanks for the information. I was able to get ignite working with RBAC
enabled K8s cluster.
The issue was with K8s calico networking, I had set it up wrongly because of
which I was getting errors.
Regards,
Vishwas
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite on RBAC enabled K8s cluster
Posted by Roman Guseinov <ro...@gromtech.ru>.
Hi Vishwas,
Where your kubernetes cluster is located? Google Cloud, Openshift or
somewhere else?
I've checked your service account configs (with role and binding) on Google
Cloud. It works well. Maybe the problem in your deployment file or in
changes related to docker image.
Here are the steps to deploy Ignite 2.5 on Google Cloud:
1. Grant cluster-admin role to current google user (to allow create roles):
$ kubectl create clusterrolebinding myname-cluster-admin-binding \
--clusterrole=cluster-admin \
--user=
2. Create service account and grant permissions:
$ kubectl create -f sa.yaml
$ kubectl create -f role.yaml
$ kubectl create -f rolebind.yaml
3. Create a grid service:
$ kubectl create -f service.yaml
4. Deploy GridGain cluster:
$ kubectl create -f grid.yaml
YAML-files are attached. Please try to use them.
sa.yaml <http://apache-ignite-users.70518.x6.nabble.com/file/t1601/sa.yaml>
role.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/role.yaml>
rolebind.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/rolebind.yaml>
service.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/service.yaml>
grid.yaml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1601/grid.yaml>
I suggest firstly make it works and then make changes you need. Also, please
make sure that your namespace is really `default`.
Best Regards,
Roman
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite on RBAC enabled K8s cluster
Posted by vbm <bm...@gmail.com>.
Hi Roman,
Thanks for the response. I followed the steps you mentioned and also the
below link
https://stackoverflow.com/questions/50817940/failed-to-retrieve-ignite-pods-ip-addresses/50818842#50818842
apiVersion: v1
kind: ServiceAccount
metadata:
name: ignite
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ignite-endpoint-access
namespace: default
labels:
app: ignite
rules:
- apiGroups: [""]
resources: ["endpoints"]
resourceNames: ["ignite"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ignite-role-binding
namespace: default
labels:
app: ignite
subjects:
- kind: ServiceAccount
name: ignite
roleRef:
kind: Role
name: ignite-endpoint-access
apiGroup: rbac.authorization.k8s.io
I have added, serviceAccount: ignite in deployment spec.
But still I am facing issues, I have done the deployment with 2 pods. Both
of them are in running state, but they are not able to discover one another.
Below is the error trace:
16:37:06,185][SEVERE][tcp-disco-ip-finder-cleaner-#4][TcpDiscoverySpi]
Failed to clean IP finder up.
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite
pods IP addresses.
at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1810)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1921)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1896)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.UnknownHostException:
kubernetes.default.svc.cluster.local
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
at
sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
The only difference is that the ignite server in the container is running
with non root user. I have created a non root user and started the process
with that user.
Not sure if that makes a difference ?
Regards,
Vishwas
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite on RBAC enabled K8s cluster
Posted by Roman Guseinov <ro...@gromtech.ru>.
Hi Vishwas,
Usually, this issue happens because of:
1. Default service account doesn't have permissions to use the endpoint to
get all related IP addresses (of Ignite nodes).
2. A cluster is deployed in not default namespace. For example, in Openshift
namespace is a project name.
To resolve that you need to create a service account and grant necessary
permissions (we don't need cluster-admin role):
*sa.yml*
*role.yml*
*rolebind.yml*
Do not forget to replace <namespace>, add `serviceAccountName` into
DeploymentConfig [1] and specify namespace property of
TcpDiscoveryKubernetesIpFinder [1].
Best Regards,
Roman
[1]
https://apacheignite.readme.io/docs/kubernetes-deployment#section-deploying-on-openshift
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/