[I] Multi-node testkit: Controller blocks on a Netty future inside its message handler, and the conductor port defaults to a fixed 4711 [pekko]

["He-Pin (via GitHub)" <gi...@apache.org>]

He-Pin opened a new issue, #3693:
URL: https://github.com/apache/pekko/issues/3693

   ## Summary
   
   Two related weaknesses in the multi-node testkit's conductor setup:
   
   1. `Controller` answers `GetSockAddr` by calling `Netty`'s blocking `ChannelFuture.sync()` inside its message handler, so a bind that is slow or has failed blocks (or throws on) the thread of the actor that also hosts the `BarrierCoordinator`.
   2. `MultiNodeSpec` defaults the conductor port to a fixed `4711`, so two multi-node runs on the same machine — or one leftover process — collide, and the failure surfaces far from its cause.
   
   ## Evidence
   
   ### Blocking `.sync()` in the message handler
   
   `multi-node-testkit/src/main/scala/org/apache/pekko/remote/testconductor/Conductor.scala:530-532`:
   
   ```scala
       case GetNodes    => sender() ! nodes.keys
       case GetSockAddr =>
         sender() ! connection.channelFuture.sync().channel().localAddress()
     }
   ```
   
   `connection` is created in the actor's constructor body, `:436-446`:
   
   ```scala
   private[pekko] class Controller(private var initialParticipants: Int, controllerPort: InetSocketAddress) extends Actor {
     ...
     val connection: RemoteConnection = RemoteConnection(
       Server,
       controllerPort,
       settings.ServerSocketWorkerPoolSize,
       new ConductorHandler(settings.QueryTimeout, self, Logging(context.system, classOf[ConductorHandler])))
   ```
   
   with the bind itself at `RemoteConnection.scala:162`. `sync()` rethrows a bind failure, so an address already in use becomes an exception thrown out of `Controller`'s receive. The same actor also owns the barrier coordinator, `Conductor.scala:468`:
   
   ```scala
     val barrier = context.actorOf(Props[BarrierCoordinator](), "barriers")
   ```
   
   `Controller`'s `supervisorStrategy` (`:454-461`) only decides on barrier-specific exceptions (`BarrierTimeout`, `FailedBarrier`, `BarrierEmpty`, `WrongBarrier`, `ClientLost`, `DuplicateNode`), so a bind failure is not covered by an explicit directive.
   
   ### Fixed default port
   
   `multi-node-testkit/src/main/scala/org/apache/pekko/remote/testkit/MultiNodeSpec.scala:227-232`:
   
   ```scala
      * -Dmultinode.server-port=4711
      ...
     val serverPort: Int = Integer.getInteger("multinode.server-port", 4711)
   
     require(serverPort > 0 && serverPort < 65535, "multinode.server-port is out of bounds: " + serverPort)
   ```
   
   `:479-485`:
   
   ```scala
     private val controllerAddr = new InetSocketAddress(serverName, serverPort)
     ...
         if (selfIndex == 0) tc.startController(initialParticipants, myself, controllerAddr)
         else tc.startClient(myself, controllerAddr)
   ```
   
   Note that `require` rejects `0`, so the ephemeral-port option is not currently reachable through this setting.
   
   Port-0 support already exists on the controller side — `Conductor.scala:103-110` documents it and resolves the real address by asking the controller:
   
   ```scala
     /**
      * Obtain the port to which the controller’s socket is actually bound. This
      * will deviate from the configuration in `pekko.testconductor.port` in case
      * that was given as zero.
      */
     def sockAddr: Future[InetSocketAddress] = {
       import Settings.QueryTimeout
       (controller ? GetSockAddr).mapTo(classTag[InetSocketAddress])
     }
   ```
   
   and `startController` (`:89-99`) already chains the resolved address into `startClient`. What is missing is a way for node 0, which runs in its own JVM, to publish the chosen port to the other nodes before they call `startClient` with a pre-computed `controllerAddr`.
   
   ## Impact
   
   A port collision does not fail fast with "address already in use". Instead the non-controller nodes attempt to reach a conductor that is not there and eventually time out on their queries or barriers, which reads as an unexplained hang or a barrier failure in whichever spec lost the race. Because the port is fixed, this is reproducible by running two multi-node suites concurrently on one machine, or by a previous run leaving a socket in `TIME_WAIT`.
   
   Blocking on `sync()` additionally means a slow bind stalls every barrier in the suite, since `Controller` and `BarrierCoordinator` share the actor.
   
   ## Suggested direction
   
   - Replace the blocking `sync()` with an asynchronous completion: register a listener on the bind future and answer `GetSockAddr` from the callback, or move the bind out of the constructor into `preStart` and pipe the result to self so a failure becomes an explicit message that can stop the actor with a clear reason instead of throwing out of a message handler.
   - Allow `multinode.server-port=0` (relax the `require` at `:232`) and have node 0 print the resolved port in a machine-readable form that the multi-node runner can capture and pass to the remaining nodes as a system property. `project/MultiNode.scala` is where the runner is wired.
   - Consider failing the whole run immediately and legibly when the conductor cannot bind, rather than letting the other nodes time out.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org