RE: Apache Geode behind NAT firewall

[Rupert St John Webster <ru...@impress-solutions.com>]

Hi Wai Lun,

From here<https://serverfault.com/questions/308662/how-do-i-fix-a-failed-to-retrieve-rmiserver-stub-jmx-error/357460#357460> it could be that the initial JMX connection is made on port 1099 but the JMX host then picks "dynamic" ports for later traffic, which are blocked.
That and a blog<http://hirt.se/blog/?p=289> on "Tunneling JMX" suggests that the RMI server hostname and port need to be specified in the JDK.

It looks like this is the answer<https://serverfault.com/a/738767/212217> to set a specific port for JMX


-Dcom.sun.management.jmxremote.port=8084

-Dcom.sun.management.jmxremote.rmi.port=8084

There's also another setting<https://stackoverflow.com/questions/10173834/java-rmi-djava-rmi-server-hostname-localhost-still-opens-a-socket-listening-on>  Djava.rmi.server.hostname

From the Wireshark it looks to me like outbound traffic on the "dynamic" JMX ports is blocked.
There are TCP packets from ports 54964, 60498 and 60499 to port 1099 followed by ICMP message<https://superuser.com/questions/1044361/why-icmp-is-different-that-tcp-and-udp> that a destination is unreachable.

[cid:image001.png@01D7BC38.7E97B160]

[cid:image003.png@01D7BC39.A7DB64D0]

My guess is that the 54964, 60498 and 60499 ports are the "dynamic" ports used by the JMX manager.
At the same time it looks like port 54969 for some kind of RMI ping is getting through to port 1099.

[cid:image002.png@01D7BC39.15DECE00]

Finally, have you tried server bindings to the A.B.C.D firewall box?

Cheers.

From: Wai Lun Poon <wa...@impress-solutions.com>
Sent: 06 October 2021 17:10
To: user@geode.apache.org
Cc: Rupert St John Webster <ru...@impress-solutions.com>; Michael Poon <mi...@impress-solutions.com>; Pravin Dave <pr...@impress-solutions.com>
Subject: Apache Geode behind NAT firewall

Hi,

We have a Geode locator and server running on a machine that is behind a NAT firewall. I have replaced the firewall's IP address with A.B.C.D and the internal IP address of the machine running the Geode locator with W.X.Y.Z.

Previously the machine running Geode had only Windows Defender Firewall enabled and an inbound rule set to allow traffic to ports 10334, 1099 and 40404 from remote IP addresses which we whitelisted. This setup allowed us to connect to the Geode locator from those remote IP addresses that were whitelisted.

However, once we placed the same machine behind the NAT firewall and configured the same rule we set up under Windows Firewall, we can no longer connect to the locator from the remote IP addresses whitelisted. The IP addresses we whitelisted are for machines outside of the firewall.

For example, when we tried connecting to the locator through gfsh, it gave us a java connection exception as shown below in Figure 1. It appears it was able to connect to the locator running on 10334 but failed to do so for the JMX manager on port 1099 using the internal IP address of the machine running the Geode locator.

On the second try, we tried specifying the firewall's IP address for the JMX manager tag but got a slightly different connection exception shown in Figure 1.

We also ran two Wireshark captures from the whitelisted IP address on port 1099 for one Geode locator that was behind the Window's Firewall only  in Figure 2 and the other one behind the NAT firewall (A.B.C.D) in Figure 3. We noticed on the capture for the NAT firewall, it wasn't able to establish a RMI stream which we think is the cause of the exception given on gfsh.

Do we need to start the locator with specific settings to get this to work or is this related to allowing RMI traffic/stream on the NAT firewall?  Please find the settings that the Geode locator was started with in Figure 4. In the Gemfire properties file we have the server-bind-address and jmx-manager-bind-address tags set to the internal IP address of the machine (W.X.Y.Z).


Figure 1
[cid:image006.png@01D7BC37.F7BFBC00]

Figure 2 - Wireshark capture from remote machine to Geode locator behind Windows Defender Firewall

[cid:image007.png@01D7BC37.F7BFBC00]

Figure 3 - Wireshark capture from remote machine to Geode locator behind NAT firewall

[cid:image008.png@01D7BC37.F7BFBC00]

Figure 4
[cid:image009.png@01D7BC37.F7BFBC00]

Kind regards,

Wai Lun Poon
System Analyst

[ISL_logo-V2]

[cid:image011.png@01D7BC37.F7BFBC00]<http://www.facebook.com/ImpressSolutions> [Description: Description: Description: http://www.impress-solutions.com/emailsig/twitter_scaled.png] <http://www.twitter.com/ImpressUK>  [Description: Description: Description: http://www.impress-solutions.com/emailsig/LinkedIn_scaled.png] <http://www.linkedin.com/company/impress-solutions>

Tel: 01708 759 760
Fax: 01708 759 761
Email:  wailun.poon@impress-solutions.com<ma...@impress-solutions.com>
Website: www.impress-solutions.co.uk
Head Office address: 3 Holgate Court, 4-10 Western Road, Romford, Essex, RM1 3JS
City address: City Point,  1 Ropemaker Street,  17th Floor, Moorgate, London, EC2Y 9HT


[BSI-Assurance-Mark-ISO-9001-2015-KEYB]

ISO 9001 Certified by BSI Group. Certificate Number: FS 653755

Please Note:
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any miss-transmission.
If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender.
You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient.
Impress Solutions Ltd and any of its subsidiaries each reserve the right to monitor all e-mail communications through its networks.

Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them to be the views of any such entity.

Registered office address: 3 Holgate Court, 4-10 Western Road, Romford, Essex, RM1 3JS
Registration Number 03412238