You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/02/01 21:22:23 UTC
[GitHub] [cloudstack] nxsbi opened a new issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
nxsbi opened a new issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and master branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Kubernetes Service
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on master branch.
-->
~~~
4.15.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Base install of 4.15.0 (upgraded from 4.11)
Kubernetes Service enabled
CoreOS template
community Kubernetes ISO (tried multiple from v 1.11.4 to 1.16.3)
Using Advanced Networking
User account uses Isolated Network (not L2 or Shared)
SSL is enabled for CS GUI, and System VMs
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
CentOS 7 for Management Server
##### SUMMARY
<!-- Explain the problem/feature briefly -->
On a freshly upgraded version to Cloudstack 4.15 (from 4.11), when I create Kubernetes Cluster (regardless of which version), the master and worker VMs are getting created and running successfully, but after the Timeout setting (default 3600 seconds) expires I see the state - "Error". Further more, under the "Access" tab, I see "Kubernetes cluster kubeconfig not available currently". I cannot download the config file/never becomes available.
This happens in Isolated Networks with source NAT enabled. I also tested on a Shared Network on a VLAN directly on the router.
It seems the VMs are getting setup but something is getting blocked when trying to check the status of the service. I have opened all ports for egress in the Isolated Network. ( I can see the data load of 200+MB taking place on the master and worker node via CS GUI)
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Kubernetes Service should show as Active
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
Kubernetes Service Shows as Error
~~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771239111
After further digging... I logged into the master node using SSH via Private Key, downloaded the /etc/kubernetes/kubectl.conf and saved it locally as kube.conf
I ran a few commands. However, the Dashboard is still not visible.
I saw the Issue #4146
It mentions about running -- curl -k on management server I think -- However in my case, management server is on a separate VLAN and the users will never have that kind of access. I ran the curl command inside the master node, and it does look like a certificate issue.. see screenshot below.
I do not know what else to check.
At this point, this is a show stopper for using Kubernetes Service for us, and is a critical bug in my opinion. I hope there is some workaround (that continues to work when new clusters get created)!!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-774253171
@ravening Let me explain a but further to provide context for anyone that reads this thread in the future --
My network setup uses a hardware firewall and router as the first device to internet connection.
All VLANs used for Management and Public access are defined here, not Guest VLANs
That connects to switches. In the Switches, all VLANs are defined and tagged.
The switches then connect to the Virtualization Servers.
The Management VLAN has the CS server, independently running in the virtualization server, on its own VLAN, which is defined in the Router and all switches. IF The Management server is running on Default VLAN (VLAN 1), you will not run into this issue. I tested it that way, and it works. But we have to use VLAN for management servers (per internal policy)
Prior to Kubernetes, there was no reason for the Management VLAN to have direct access to any Public VLAN. Management server does all work by connecting to VR via the Virtualization Host Server using the Link Local IP.
So in this case, the Management Server was trying to communicate to the Public IP of the Kubernetes cluster Network (that forwards the traffic to the Master node). In my case, the Public IP is just another VLAN and not an Internet accessible IP (not true public). So in my case this is inter VLAN traffic, and that went to the Router to get routed from one VLAN to another, This is what was getting blocked. I did see entries in the firewall log after resolving this issue.
So my change was:
Allow Traffic from the IP of the Cloudstack Server to Any VLAN on Port 6443.
The Reverse traffic is blocked. So no VR can directly reach the cloudstack server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773131690
> @shwstppr Per your last comment - Management Server needs to be able to SSH to VMs through VR.
> That means Management Server needs to be able to connect to VR (and vice versa).
> FYI - I have limited understanding of how that communication needs to happen, and I am trying to learn here, so excuse my silly question...
>
> If the Management server sits on its own VLAN with a single NIC, the VR is on its own VLAN (its created as an Isolated Network by default, which gets its own VLAN) , Both have Internet connectivity, but the VR is not exposed (meaning the Public IP assigned is just another VLAN), how would they ever be able to communicate? Secondly, dosen't that introduce a huge security risk if the network is accessible from the VR (and hence any VM on that VR) to Management server?
>
> Again, I do not know if my assumption here is completely off, so please correct/explain as needed
>
> EDIT ---- I Think I answered my own question after some more research... The Management server connects to the Virtualization Host (XCP-ng in my case), and uses the "ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@LinkLocal" to get into the VR....
>
> NOTE #2 -- In #4639 I added more details of my testing with the new build you provided. However it still failed. Here is a link for ease -- [#4639 (comment)](https://github.com/apache/cloudstack/pull/4639#issuecomment-773014094)
@nxsbi as far as I know, when kubernetes cluster is created, some port forwarding rules are added for vms in the cluster. for example,
VR public IP:2222 -> master
VR public IP:2223 -> node-1
VR public IP:2224 -> node-2
mgt server connects to kubernetes master/nodes via the VR public IP and ports above, not linklocal IP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi closed issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi closed issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773486212
@shwstppr That was it!! The Management Server was not able to reach the Virtual router on port 2222 via ssh.
Turned out, the main Hardware firewall (outside of Cloudstack) does not allow inter VLAN communication as a default. That is what was blocking it. I added rule to allow the communication, and was able to create a cluster just now!
Thanks so much
Perhaps we need to add a note in the documentation that for Kubernetes cluster creation, IF the Management server setup uses a VLAN for its networking, appropriate Firewall rules need to be added to allow communication from management VLAN to all VLANs used for Public traffic where Kubernetes Cluster could be created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773131690
> @shwstppr Per your last comment - Management Server needs to be able to SSH to VMs through VR.
> That means Management Server needs to be able to connect to VR (and vice versa).
> FYI - I have limited understanding of how that communication needs to happen, and I am trying to learn here, so excuse my silly question...
>
> If the Management server sits on its own VLAN with a single NIC, the VR is on its own VLAN (its created as an Isolated Network by default, which gets its own VLAN) , Both have Internet connectivity, but the VR is not exposed (meaning the Public IP assigned is just another VLAN), how would they ever be able to communicate? Secondly, dosen't that introduce a huge security risk if the network is accessible from the VR (and hence any VM on that VR) to Management server?
>
> Again, I do not know if my assumption here is completely off, so please correct/explain as needed
>
> EDIT ---- I Think I answered my own question after some more research... The Management server connects to the Virtualization Host (XCP-ng in my case), and uses the "ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@LinkLocal" to get into the VR....
>
> NOTE #2 -- In #4639 I added more details of my testing with the new build you provided. However it still failed. Here is a link for ease -- [#4639 (comment)](https://github.com/apache/cloudstack/pull/4639#issuecomment-773014094)
@nxsbi as far as I know, when kubernetes cluster is created, some port forwarding rules are added for vms in the cluster. for example,
VR public IP:2222 -> master
VR public IP:2223 -> node-1
VR public IP:2224 -> node-2
mgt server connects to kubernetes master/nodes via the VR public IP and ports above, not linklocal IP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771497622
@weizhouapache @ravening can you please comment, this seems the same issue mentioned in #4146
@nxsbi I've tried to reproduce this in my test environment with SSL enabled for GUI but I'm not being able to reproduce. Must be the difference in certificates
The problem seems to be at https://github.com/apache/cloudstack/blob/ff376d8187ec2687fef06c611740e8d4befba6e7/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java#L223 with server return 403 response.
Not sure if using HttpsURLConnection will help. I've created a PR (#4639) against 4.14, can you guys please help to test it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773489306
IF the Management server setup uses a VLAN for its networking, appropriate Firewall rules need to be added to any router/hardware firewall to allow communication from management VLAN to all VLANs used for Public traffic where Kubernetes Cluster could be created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-772405558
@ravening yes, management server should be able to SSH to VMs through VR.
Service adds firewall and port forwarding rule for that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771497622
@weizhouapache @ravening can you please comment, this seems the same issue mentioned in #4146
@nxsbi I've tried to reproduce this in my test environment with SSL enabled for GUI but I'm not being able to reproduce. Must be the difference in certificates
The problem seems to be at https://github.com/apache/cloudstack/blob/master/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java#L223 with server return 403 response.
Not sure if using HttpsURLConnection will help. I've created a PR (#4639) against 4.14, can you guys please help to test it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773004938
@shwstppr Per your last comment - Management Server needs to be able to SSH to VMs through VR.
That means Management Server needs to be able to connect to VR (and vice versa).
FYI - I have limited understanding of how that communication needs to happen, and I am trying to learn here, so excuse my silly question...
If the Management server sits on its own VLAN with a single NIC, the VR is on its own VLAN (its created as an Isolated Network by default, which gets its own VLAN) , Both have Internet connectivity, but the VR is not exposed (meaning the Public IP assigned is just another VLAN), how would they ever be able to communicate? Secondly, dosen't that introduce a huge security risk if the network is accessible from the VR (and hence any VM on that VR) to Management server?
Again, I do not know if my assumption here is completely off, so please correct/explain as needed
EDIT ---- I Think I answered my own question after some more research... The Management server connects to the Virtualization Host (XCP-ng in my case), and uses the "ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@LinkLocal" to get into the VR....
NOTE #2 -- In #4639 I added more details of my testing with the new build you provided. However it still failed. Here is a link for ease -- https://github.com/apache/cloudstack/pull/4639#issuecomment-773014094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773004938
@shwstppr Per your last comment - Management Server needs to be able to SSH to VMs through VR.
That means Management Server needs to be able to connect to VR (and vice versa).
FYI - I have limited understanding of how that communication needs to happen, and I am trying to learn here, so excuse my silly question...
If the Management server sits on its own VLAN with a single NIC, the VR is on its own VLAN (its created as an Isolated Network by default, which gets its own VLAN) , Both have Internet connectivity, but the VR is not exposed (meaning the Public IP assigned is just another VLAN), how would they ever be able to communicate? Secondly, dosen't that introduce a huge security risk if the network is accessible from the VR (and hence any VM on that VR) to Management server?
Again, I do not know if my assumption here is completely off, so please correct/explain as needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771885885
@shwstppr I asked on the other thread (#4639) - but how do I test this change in my environment?
I am not a developer, so I am not sure..
Please advise
PS> When you tried to reproduce, did you place the Management server on separate VLAN and the Kubernetes cluster in a separate VLAN such that they cannot talk to each other? When I tested with the 4.15 RC2, I did not put them on separate VLAN, and it worked fine. I am wondering if that has anything to do with it (my RC2 environment was already wiped).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773456915
@nxsbi Service uses SSH over ports 2222 to 2222+n. SSH on worker nodes is done only during k8s version upgrade.
Firewall and port forwarding rules must be automatically provisioned by the service in the cluster's network.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771885885
@shwstppr I asked on the other thread (#4639) - but how do I test this change in my environment?
I am not a developer, so I am not sure..
Please advise
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773450124
@shwstppr But my management server is able to curl -k the public IP. It fails on curl only without -k option.
So this should be working right? Do I need any other port besides 6443 open? Does it communicate in any other way besides https?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773450124
@shwstppr But my management server is able to curl -k the public IP. It fails on curl only without -k option.
So this should be working right? Do I need any other port besides 6443 open? Does it communicate in any other way besides https?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773198073
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi closed issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi closed issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771239111
After further digging... I logged into the master node using SSH via Private Key, downloaded the /etc/kubernetes/kubectl.conf and saved it locally as kube.conf
I ran a few commands. However, the Dashboard is still not visible.
I saw the Issue #4146
It mentions about running -- curl -k on management server I think -- However in my case, management server is on a separate VLAN and the users will never have that kind of access. I ran the curl command inside the master node, and it does look like a certificate issue.. see screenshot below.
I do not know what else to check.
At this point, this is a show stopper for using Kubernetes Service for us, and is a critical bug in my opinion. I hope there is some workaround (that continues to work when new clusters get created)!!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-788675052
Cert check/SSL issue fixed in https://github.com/apache/cloudstack/pull/4639
Please re-open if we something else was missed (docs?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773198073
@nxsbi as correctly mentioned by @weizhouapache Kubernetes service connects cluster's master/nodes via the VR **public IP**. This is one of the requirement right now.
#4639 just tries to fix SSL validation issue while trying to access API server of k8s cluster.
From your logs shared in the PR, it seems management server is not being able to reach master VM through VR's public IP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771497622
@weizhouapache @ravening can you please comment, this seems the same issue mentioned in #4146
@nxsbi I've tried to reproduce this in my test environment with SSL enabled for GUI but I'm not being able to reproduce. Must be the difference in certificates
The problem seems to be at https://github.com/apache/cloudstack/blob/ff376d8187ec2687fef06c611740e8d4befba6e7/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java#L223 with server return 403 response.
Not sure if using HttpsURLConnection will help. I've created a PR (#4639) against 4.14, can you guys please help to test it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
shwstppr edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771497622
@weizhouapache @ravening can you please comment, this seems the same issue mentioned in #4146
@nxsbi I've tried to reproduce this in my test environment with SSL enabled for GUI but I'm not being able to reproduce. Must be the difference in certificates
The problem seems to be at https://github.com/apache/cloudstack/blob/master/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java#L223 with server return 403 response.
Not sure if using HttpsURLConnection will help. I've created a PR (#4639) against 4.14, can you guys please help to test it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-774104725
> @shwstppr That was it!! The Management Server was not able to reach the Virtual router on port 2222 via ssh.
> Turned out, the main Hardware firewall (outside of Cloudstack) does not allow inter VLAN communication as a default. That is what was blocking it. I added rule to allow the communication, and was able to create a cluster just now!
>
> Thanks so much
>
> Perhaps we need to add a note in the documentation that for Kubernetes cluster creation, IF the Management server setup uses a VLAN for its networking, appropriate Firewall rules need to be added to allow communication from management VLAN to all VLANs used for Public traffic where Kubernetes Cluster could be created.
@nxsbi can you let me know which rules you added?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-774253171
@ravening Let me explain a but further to provide context for anyone that reads this thread in the future --
My network setup uses a hardware firewall and router as the first device to internet connection.
All VLANs used for Management and Public access are defined here, not Guest VLANs
That connects to switches. In the Switches, all VLANs are defined and tagged.
The switches then connect to the Virtualization Servers.
The Management VLAN has the CS server, independently running in the virtualization server, on its own VLAN, which is defined in the Router and all switches. IF The Management server is running on Default VLAN (VLAN 1), you will not run into this issue. I tested it that way, and it works. But we have to use VLAN for management servers (per internal policy)
Prior to Kubernetes, there was no reason for the Management VLAN to have direct access to any Public VLAN. Management server does all work by connecting to VR via the Virtualization Host Server using the Link Local IP.
So in this case, the Management Server was trying to communicate to the Public IP of the Kubernetes cluster Network (that forwards the traffic to the Master node). Since this is inter VLAN traffic, the traffic went to the Router, This is what was getting blocked. I did see entries in the firewall log.
So my change was:
Allow Traffic from the IP of the Cloudstack Server to Any VLAN on Port 6443.
The Reverse traffic is blocked. So no VR can directly reach the cloudstack server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771205646
Additional Information:
I tried to get Config file via CMK.
I get error "HTTP 530 error code 9999"
Are there any workarounds to make Kubernetes work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
ravening commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-772400626
@shwstppr Just to confirm the requirements, the mgt server should be able to ssh to vm's?
Im planning to test this again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771885885
@shwstppr I asked on the other thread (#4639) - but how do I test this change in my environment?
I am not a developer, so I am not sure..
Please advise
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-771885885
@shwstppr I asked on the other thread (#4639) - but how do I test this change in my environment?
I am not a developer, so I am not sure..
Please advise
PS> When you tried to reproduce, did you place the Management server on separate VLAN and the Kubernetes cluster in a separate VLAN such that they cannot talk to each other? When I tested with the 4.15 RC2, I did not put them on separate VLAN, and it worked fine. I am wondering if that has anything to do with it (my RC2 environment was already wiped).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi commented on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi commented on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773450124
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nxsbi edited a comment on issue #4637: Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network
Posted by GitBox <gi...@apache.org>.
nxsbi edited a comment on issue #4637:
URL: https://github.com/apache/cloudstack/issues/4637#issuecomment-773450124
@shwstppr But my management server is able to curl -k the public IP. It fails on curl only without -k option.
So this should be working right? Do I need any other port besides 6443 open? Does it communicate in any other way besides https?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org