[jira] [Resolved] (LIVY-1086) Fix macOS unit test failures by binding Spark driver host in test fixtures

["Gyorgy Gal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gyorgy Gal resolved LIVY-1086.
------------------------------
    Resolution: Fixed

> Fix macOS unit test failures by binding Spark driver host in test fixtures
> --------------------------------------------------------------------------
>
>                 Key: LIVY-1086
>                 URL: https://issues.apache.org/jira/browse/LIVY-1086
>             Project: Livy
>          Issue Type: Bug
>            Reporter: Nilesh Rathi
>            Assignee: Nilesh Rathi
>            Priority: Major
>             Fix For: 1.0.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> h2. Problem
> Follow-up to LIVY-1065 (merged in [PR #538|https://github.com/apache/livy/pull/538]).
> PR #538 pinned Livy, RSC, YARN mini-cluster, and several unit-test services to loopback ({{{}127.0.0.1{}}}) via {{TestUtils.TEST_BIND_HOST}} so tests pass on macOS. However, several unit tests still start Spark directly without setting {{spark.driver.host}} and {{{}spark.driver.bindAddress{}}}.
> On macOS, Spark's driver can advertise the machine's LAN IP (from {{{}InetAddress.getLocalHost(){}}}), which peer processes on the same host cannot reach. This causes failures such as:
>  * {{RSCClient instance stopped}} and connect timeouts to the LAN IP (e.g. {{{}Failed to connect to /192.168.x.x:...{}}})
>  * Spark stage failures / {{RemoteClassLoaderError}} when executors cannot reach the driver
>  * Timeouts in REPL, RSC, server interactive, and thriftserver session tests
> Linux CI behavior is unchanged ({{{}127.0.0.1{}}} is equally valid there).
> h2. Proposed fix
> Extend the LIVY-1065 approach by setting {{spark.driver.host}} and {{spark.driver.bindAddress}} to {{TestUtils.TEST_BIND_HOST}} in test fixtures that create Spark sessions/drivers directly.
> Add {{TestUtils.SPARK_DRIVER_HOST}} and {{TestUtils.SPARK_DRIVER_BIND_ADDRESS}} constants in {{client-common/TestUtils.java}} to avoid duplicating raw config key strings.
> Fixtures/modules affected include (non-exhaustive):
>  * {{{}repl{}}}: {{{}BaseSessionSpec{}}}, {{{}SessionSpec{}}}, {{{}ReplDriverSuite{}}}, interpreter specs
>  * {{{}rsc{}}}: {{TestSparkClient}}
>  * {{{}scala-api{}}}: {{ScalaClientTest}}
>  * {{{}server{}}}: {{{}BaseInteractiveServletSpec{}}}, {{InteractiveSessionSpec}}
>  * {{{}thriftserver{}}}: {{{}ThriftServerBaseTest{}}}, {{{}ThriftSessionTest{}}}, {{ColumnBufferTest}}
>  * {{{}integration-test{}}}: {{MiniCluster}} (use {{TestUtils.SPARK_DRIVER_HOST}} constant)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)