You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Tejas Sheth <ts...@gmail.com> on 2017/10/06 08:37:28 UTC
DNS resolution issue in cloudstack tenant VM
Hello,
We started facing strange issue with cloudstack VM. where all the VMs in
one particular tenant are not able to resolve DNS. since we are using
advanced networking we have tried to reboot virtual router and other DNS
services. still that particular tenant is not able to resolve the DNS.
following troubleshooting steps completed.
1) From the tenant (with issue) we are able reach DNS server with ICMP
(ping)
2) from other tenant we are able to reach DNS server on ICMP. also DNS
resolution working from other tenant.
3) We have checked ingress and egress traffic settings where we have
allowed all inbound and outbound but still DNS resolution is not working.
NOTE: DNS server is currently hosted in public network subnet which is
being used by all the tenants
Re: DNS resolution issue in cloudstack tenant VM
Posted by Tejas Sheth <ts...@gmail.com>.
We have enabled "Bypass internal dns, use external dns1 and dns2" from the
zone settings and it started resolving.
Simon and Dag, Thank you.
Cheers!!
On Sat, Oct 7, 2017 at 11:39 AM, Tejas Sheth <ts...@gmail.com> wrote:
> Hello Dag Sonstebo,
>
> First of all thanks for the detailed reply.
> Just wanted to update you on following.
>
> 1) Any reason why you aren’t just letting the clients use the VR for DNS
> forwarding, rather than going direct?
>
> --> in fact, we are letting clients to use VR for DNS forwarding . All of
> the tenant VMs are using VR ip as DNS and VR is forwarding DNS to external
> DNS.
>
> it's working for all other tenant (account) except one where we are
> facing issue.
>
> Any idea what can be the cause of it?
>
>
> I am also working on Simon's input.
>
> Thanks,
> Tejas
>
> On Oct 6, 2017 6:03 PM, "Simon Weller" <sw...@ena.com.invalid> wrote:
>
> The other thing to check is that you haven't specified your external DNS
> as internal DNS within the advanced zone settings. I know from experience
> that CloudStack will place a static route into the VR to force traffic out
> the internal interface if you specify internal dns, since the default route
> is external. This will cause asymmetry and break things, but ICMP will
> continue to work.
>
>
> - Si
>
>
> ________________________________
> From: Dag Sonstebo <Da...@shapeblue.com>
> Sent: Friday, October 6, 2017 4:01 AM
> To: users@cloudstack.apache.org
> Subject: Re: DNS resolution issue in cloudstack tenant VM
>
> Hi Tejas,
>
> “DNS server is currently hosted in public network subnet which is being
> used by all the tenants”.
>
> Is this DNS server hosted directly on the public network and externally to
> CloudStack, or is it hosted on a CloudStack isolated/VPC network with DNS
> services port forwarded to the public network?
>
> If the latter then we have seen a few issues around “hairpin NATing” –
> where VMs on one isolated network isn’t able to access services on another
> isolated network over the common public network. This has been found to be
> down to order of Iptables rules on the VR. There were a few PRs to fix this
> issue earlier in the summer – and I believe those fixes have been included
> in 4.9.3.
>
> If the former – i.e. you are simply hosting a DNS server directly on the
> public network then I haven’t seen this before, I would suggest doing some
> packet sniffing to see what is going on on the network.
>
> A couple of obvious ones which you have probably checked:
> - Is the VR actually handing out the correct DNS settings to the clients?
> If not it could be the DNSmasq DHCP service is unhappy about something.
> - Any reason why you aren’t just letting the clients use the VR for DNS
> forwarding, rather than going direct?
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 06/10/2017, 09:37, "Tejas Sheth" <ts...@gmail.com> wrote:
>
> Hello,
>
> We started facing strange issue with cloudstack VM. where all the
> VMs in
> one particular tenant are not able to resolve DNS. since we are using
> advanced networking we have tried to reboot virtual router and other
> DNS
> services. still that particular tenant is not able to resolve the DNS.
>
> following troubleshooting steps completed.
> 1) From the tenant (with issue) we are able reach DNS server with
> ICMP
> (ping)
> 2) from other tenant we are able to reach DNS server on ICMP. also
> DNS
> resolution working from other tenant.
> 3) We have checked ingress and egress traffic settings where we have
> allowed all inbound and outbound but still DNS resolution is not
> working.
>
>
> NOTE: DNS server is currently hosted in public network subnet which is
> being used by all the tenants
>
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<
> http://www.shapeblue.com/>
>
> Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
> www.shapeblue.com
> Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
> framework developed by ShapeBlue to deliver the rapid deployment of a
> standardised ...
>
>
>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
>
Re: DNS resolution issue in cloudstack tenant VM
Posted by Tejas Sheth <ts...@gmail.com>.
Hello Dag Sonstebo,
First of all thanks for the detailed reply.
Just wanted to update you on following.
1) Any reason why you aren’t just letting the clients use the VR for DNS
forwarding, rather than going direct?
--> in fact, we are letting clients to use VR for DNS forwarding . All of
the tenant VMs are using VR ip as DNS and VR is forwarding DNS to external
DNS.
it's working for all other tenant (account) except one where we are
facing issue.
Any idea what can be the cause of it?
I am also working on Simon's input.
Thanks,
Tejas
On Oct 6, 2017 6:03 PM, "Simon Weller" <sw...@ena.com.invalid> wrote:
The other thing to check is that you haven't specified your external DNS as
internal DNS within the advanced zone settings. I know from experience that
CloudStack will place a static route into the VR to force traffic out the
internal interface if you specify internal dns, since the default route is
external. This will cause asymmetry and break things, but ICMP will
continue to work.
- Si
________________________________
From: Dag Sonstebo <Da...@shapeblue.com>
Sent: Friday, October 6, 2017 4:01 AM
To: users@cloudstack.apache.org
Subject: Re: DNS resolution issue in cloudstack tenant VM
Hi Tejas,
“DNS server is currently hosted in public network subnet which is being
used by all the tenants”.
Is this DNS server hosted directly on the public network and externally to
CloudStack, or is it hosted on a CloudStack isolated/VPC network with DNS
services port forwarded to the public network?
If the latter then we have seen a few issues around “hairpin NATing” –
where VMs on one isolated network isn’t able to access services on another
isolated network over the common public network. This has been found to be
down to order of Iptables rules on the VR. There were a few PRs to fix this
issue earlier in the summer – and I believe those fixes have been included
in 4.9.3.
If the former – i.e. you are simply hosting a DNS server directly on the
public network then I haven’t seen this before, I would suggest doing some
packet sniffing to see what is going on on the network.
A couple of obvious ones which you have probably checked:
- Is the VR actually handing out the correct DNS settings to the clients?
If not it could be the DNSmasq DHCP service is unhappy about something.
- Any reason why you aren’t just letting the clients use the VR for DNS
forwarding, rather than going direct?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 06/10/2017, 09:37, "Tejas Sheth" <ts...@gmail.com> wrote:
Hello,
We started facing strange issue with cloudstack VM. where all the
VMs in
one particular tenant are not able to resolve DNS. since we are using
advanced networking we have tried to reboot virtual router and other DNS
services. still that particular tenant is not able to resolve the DNS.
following troubleshooting steps completed.
1) From the tenant (with issue) we are able reach DNS server with
ICMP
(ping)
2) from other tenant we are able to reach DNS server on ICMP. also
DNS
resolution working from other tenant.
3) We have checked ingress and egress traffic settings where we have
allowed all inbound and outbound but still DNS resolution is not
working.
NOTE: DNS server is currently hosted in public network subnet which is
being used by all the tenants
Dag.Sonstebo@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<
http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: DNS resolution issue in cloudstack tenant VM
Posted by Simon Weller <sw...@ena.com.INVALID>.
The other thing to check is that you haven't specified your external DNS as internal DNS within the advanced zone settings. I know from experience that CloudStack will place a static route into the VR to force traffic out the internal interface if you specify internal dns, since the default route is external. This will cause asymmetry and break things, but ICMP will continue to work.
- Si
________________________________
From: Dag Sonstebo <Da...@shapeblue.com>
Sent: Friday, October 6, 2017 4:01 AM
To: users@cloudstack.apache.org
Subject: Re: DNS resolution issue in cloudstack tenant VM
Hi Tejas,
“DNS server is currently hosted in public network subnet which is being used by all the tenants”.
Is this DNS server hosted directly on the public network and externally to CloudStack, or is it hosted on a CloudStack isolated/VPC network with DNS services port forwarded to the public network?
If the latter then we have seen a few issues around “hairpin NATing” – where VMs on one isolated network isn’t able to access services on another isolated network over the common public network. This has been found to be down to order of Iptables rules on the VR. There were a few PRs to fix this issue earlier in the summer – and I believe those fixes have been included in 4.9.3.
If the former – i.e. you are simply hosting a DNS server directly on the public network then I haven’t seen this before, I would suggest doing some packet sniffing to see what is going on on the network.
A couple of obvious ones which you have probably checked:
- Is the VR actually handing out the correct DNS settings to the clients? If not it could be the DNSmasq DHCP service is unhappy about something.
- Any reason why you aren’t just letting the clients use the VR for DNS forwarding, rather than going direct?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 06/10/2017, 09:37, "Tejas Sheth" <ts...@gmail.com> wrote:
Hello,
We started facing strange issue with cloudstack VM. where all the VMs in
one particular tenant are not able to resolve DNS. since we are using
advanced networking we have tried to reboot virtual router and other DNS
services. still that particular tenant is not able to resolve the DNS.
following troubleshooting steps completed.
1) From the tenant (with issue) we are able reach DNS server with ICMP
(ping)
2) from other tenant we are able to reach DNS server on ICMP. also DNS
resolution working from other tenant.
3) We have checked ingress and egress traffic settings where we have
allowed all inbound and outbound but still DNS resolution is not working.
NOTE: DNS server is currently hosted in public network subnet which is
being used by all the tenants
Dag.Sonstebo@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed by ShapeBlue to deliver the rapid deployment of a standardised ...
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: DNS resolution issue in cloudstack tenant VM
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Tejas,
“DNS server is currently hosted in public network subnet which is being used by all the tenants”.
Is this DNS server hosted directly on the public network and externally to CloudStack, or is it hosted on a CloudStack isolated/VPC network with DNS services port forwarded to the public network?
If the latter then we have seen a few issues around “hairpin NATing” – where VMs on one isolated network isn’t able to access services on another isolated network over the common public network. This has been found to be down to order of Iptables rules on the VR. There were a few PRs to fix this issue earlier in the summer – and I believe those fixes have been included in 4.9.3.
If the former – i.e. you are simply hosting a DNS server directly on the public network then I haven’t seen this before, I would suggest doing some packet sniffing to see what is going on on the network.
A couple of obvious ones which you have probably checked:
- Is the VR actually handing out the correct DNS settings to the clients? If not it could be the DNSmasq DHCP service is unhappy about something.
- Any reason why you aren’t just letting the clients use the VR for DNS forwarding, rather than going direct?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 06/10/2017, 09:37, "Tejas Sheth" <ts...@gmail.com> wrote:
Hello,
We started facing strange issue with cloudstack VM. where all the VMs in
one particular tenant are not able to resolve DNS. since we are using
advanced networking we have tried to reboot virtual router and other DNS
services. still that particular tenant is not able to resolve the DNS.
following troubleshooting steps completed.
1) From the tenant (with issue) we are able reach DNS server with ICMP
(ping)
2) from other tenant we are able to reach DNS server on ICMP. also DNS
resolution working from other tenant.
3) We have checked ingress and egress traffic settings where we have
allowed all inbound and outbound but still DNS resolution is not working.
NOTE: DNS server is currently hosted in public network subnet which is
being used by all the tenants
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue