[jira] [Created] (LIVY-1065) Bind Livy/Spark test services to 127.0.0.1 for CI reachability on loopback-only hosts

["Gabor Roczei (Jira)" <ji...@apache.org>]

Gabor Roczei created LIVY-1065:
----------------------------------

             Summary: Bind Livy/Spark test services to 127.0.0.1 for CI reachability on loopback-only hosts
                 Key: LIVY-1065
                 URL: https://issues.apache.org/jira/browse/LIVY-1065
             Project: Livy
          Issue Type: Bug
            Reporter: Gabor Roczei
            Assignee: Gabor Roczei


h2. Problem

On hosts (macOS, some CI containers) whose canonical hostname resolves to a loopback-only address, several Livy and Spark services used by the integration and unit test suites bind to the LAN IP returned by {{InetAddress.getLocalHost}}. That IP is not routable from mini-cluster YARN containers or JDBC clients running on the same host, and the tests hang or fail. Concrete symptoms:

* {{JdbcIT}} SASL handshake timeout when the client tries {{jdbc:hive2://<LAN-IP>:10090}}.
* {{RSCClient instance stopped}} caused by {{ConnectTimeoutException}} on {{/LAN-IP:10000}} — the Spark driver inside the YARN AM cannot reach the Livy RSC server.
* {{Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered}} — the driver AM advertises {{spark://YarnAM@<LAN-IP>:<port>}} and the executor container never registers.
* {{java.io.IOException: Cannot run program "Rscript"/"python": error=2, No such file or directory}} in SparkRIT and PySparkIT — YARN's default sanitised PATH omits {{/opt/homebrew/bin}} on macOS and pyenv/rbenv shims on CI runners.

h2. Fix

Force every affected bind point to {{127.0.0.1}} in the test harness, and propagate the host shell {{PATH}} into the YARN AM and executor containers.

Bind points touched:

* {{livy.server.host}}, {{livy.server.thrift.bind.host}}, {{livy.rsc.rpc.server.address}} in {{MiniLivyMain}}.
* {{spark.driver.host}}, {{spark.yarn.appMasterEnv.SPARK_LOCAL_IP}}, {{spark.executorEnv.SPARK_LOCAL_IP}} in {{MiniCluster.deploy()}}'s generated spark-defaults.conf. {{SPARK_LOCAL_IP}} is what the YARN AM driver actually reads; {{spark.driver.host}} on the submit side alone is not enough in cluster deploy mode.
* {{RSCConf.Entry.RPC_SERVER_ADDRESS = 127.0.0.1}} in every unit-test fixture that spins up an in-process RSC server: {{BaseSessionSpec}}, {{ReplDriverSuite}}, {{TestSparkClient}}, {{ScalaClientTest}}, {{BaseInteractiveServletSpec}}, {{InteractiveSessionSpec}}, {{ThriftServerBaseTest}}, {{ThriftSessionTest}}.
* {{WebServer}} bind host from {{0.0.0.0}} / {{InetAddress.getLocalHost}} to {{127.0.0.1}} in {{HttpClientSpec}} and {{LivyConnectionSpec}}, and switch the URIs those tests build to use {{server.host}} so the client and server agree on the address.

Also propagate the host shell {{PATH}} via {{spark.yarn.appMasterEnv.PATH}} / {{spark.executorEnv.PATH}} so {{RRunner}} and {{PythonRunner}} can resolve the native binaries.

h2. Scope

Test-harness / CI only — no production code paths change. The fix applies to both {{-Pspark3}} and {{-Pspark4}} matrices.

Split out from LIVY-1041 (Add support for Spark 4 via a new Maven profile) because these fixes are logically independent and were bundled there only because they surfaced most visibly while getting the Spark 4 matrix green.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)