You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/22 10:53:28 UTC

(texera) 01/01: test(workflow-core): cover the lakeFS file document against the loopback stub (#7831)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7831-fdc919110b15d358055b11cb35bc1ea4ee006dee
in repository https://gitbox.apache.org/repos/asf/texera.git

commit b7c33b0892db49a94a3b15c81eed4ebcbc12882d
Author: Xinyuan Lin <xi...@uci.edu>
AuthorDate: Sat Aug 22 10:53:06 2026 +0000

    test(workflow-core): cover the lakeFS file document against the loopback stub (#7831)
    
    ### What changes were proposed in this PR?
    
    Test-only. **No production file changes** — `git diff -- '*/src/main/*'`
    is empty.
    
    `LakeFSFileDocument` looked Docker-gated and is not:
    `LakeFSStorageClientSpec` (merged in #7273) already drives
    `LakeFSStorageClient` against an in-process `com.sun.net.httpserver`
    loopback stub, and already covers the exact two calls the document
    makes. This PR reuses that harness.
    
    Measured with `WorkflowCore/jacoco` under an identical filter for the
    before and after runs (the Docker-backed `LakeFSStorageClientMtimeSpec`
    is excluded from **both**, so these differ from the Codecov figures,
    which include it):
    
    | File | Lines | Branches |
    |---|---|---|
    | `LakeFSFileDocument.scala` | 32/80 (40.0%) → **65/80 (81.2%)** | 16/46
    → **27/46** |
    | `LakeFSStorageClient.scala` | 120/146 (82.2%) → **133/146 (91.1%)** |
    41/60 → **45/60** |
    
    **+33 lines and +11 branch arms** on the document, **+13 lines** on the
    client. Tests **38 → 49**.
    
    Covered: `asInputStream`'s presign happy path and its fallback (the
    local `def fallbackToLakeFS` lifts to `fallbackToLakeFS$1`, a counted
    method rather than an `$anonfun$`, which is why these lines move the
    number at all), `asFile` both arms including the `tempFile` memoization,
    `clear` both arms, and `LakeFSStorageClient.put`.
    
    The 15 lines still missed on the document are exactly the out-of-scope
    region — lines 133-161, the `userJwtToken`-non-empty file-service
    presign branch. Reaching it needs a forked JVM with `USER_JWT_TOKEN` set
    **and** a presign endpoint on a *fixed* port known before JVM start,
    i.e. a `build.sbt` `testGrouping` change plus a hardcodable port that
    can collide in CI. Deliberately excluded.
    
    ### Verification
    
    Review proposed 8 findings. **Six were real survivors and are now
    killed**, each proved red on a named test after the repair; two are
    genuinely unpinnable here and are recorded in the spec rather than
    papered over. Suite 47 → 49 after repair.
    
    | Survivor found by review | How it is now killed |
    |---|---|
    | both `asInputStream` call sites reduced to
    `fileRelativePath.getFileName.toString` | a nested-fixture read test
    that kills the pair *and* each site alone |
    | widening the `put` success guard to `code >= 400` | rejection test
    extended to 403/204/307; the 204 and 307 single-leg widenings each die
    separately |
    | deleting `inputStream.close()` | a recording-subclass test,
    deterministic on any platform |
    | re-downloading into the memoized `File` | `requests.clear()` plus
    `hits shouldBe empty` |
    | dropping `setChunkedStreamingMode` | the stub now records request
    headers; the put test pins `content-length: 5` and the absence of
    `transfer-encoding` |
    | `Files.exists shouldBe true` was JDK-guaranteed | now `Files.size ==
    2502` — and the mutant fails *at that line* rather than 12 lines later |
    
    **Two limits, argued rather than cargo-culted.** The fallback
    `logger.warn` at :110 cannot be pinned: `workflow-core`'s test classpath
    has no SLF4J provider at all ("No SLF4J providers were found", NOP
    loggers), and scala-logging's `isWarnEnabled` guard means the call is
    not even evaluated — a capturing appender has nothing to attach to. And
    `deleteRepo` (1 line) was left alone: it is
    `repoApi.deleteRepository(name).execute()` with no branch, so the only
    assertable fact is the generated SDK's own URL template. The existing
    spec already documents that rationale and I did not contradict it for
    one line.
    
    ### Two production bugs found while doing this, reported not fixed
    
    Both are out of scope for a test-only PR and neither is cemented by any
    new test.
    
    1. **`LakeFSFileDocument.scala:204` renders the relative path with the
    platform separator.** On a Windows worker, every nested dataset/model
    file is requested as `dir\file.csv` in the lakeFS `path` query parameter
    — 404, silent fallback, 404 again. A reviewer's suggested assertion of
    `Some("nested/dir/records.csv")` would be red against today's
    production, which is how this surfaced. Fix is a one-line `'/'`-join and
    wants its own PR.
    2. **`tempFileStream.close()` is load-bearing on Windows only.**
    Deleting it fails 4 tests here with "file is being used by another
    process" (the cleanup hits an open write handle) while shipping green
    through Linux CI. `FileOutputStream` is unbuffered, so no portable
    in-process assertion can observe it; wrapping the copy in
    `try`/`finally` is production work.
    
    ### Any related issues, documentation, discussions?
    
    Closes #7830
    
    ### How was this PR tested?
    
    ```
    sbt "WorkflowCore/testOnly org.apache.texera.amber.core.storage.model.LakeFSFileDocumentSpec org.apache.texera.amber.core.storage.util.LakeFSStorageClientSpec"
    ```
    
    ```
    [info] Suites: completed 2, aborted 0
    [info] Tests: succeeded 49, failed 0, canceled 0, ignored 0, pending 0
    ```
    
    `scalafmtCheckAll` and `scalafixAll --check` both pass.
    `LakeFSFileDocumentSpec` gains `@NonParallelTest` so it gets its own
    forked JVM, matching the sibling spec.
    
    ### Was this PR authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 5)
    
    ---------
    
    Signed-off-by: Xinyuan Lin <xi...@uci.edu>
    Co-authored-by: Copilot Autofix powered by AI <17...@users.noreply.github.com>
---
 .../storage/model/LakeFSFileDocumentSpec.scala     | 309 ++++++++++++++++++++-
 .../storage/util/LakeFSStorageClientSpec.scala     | 113 +++++++-
 2 files changed, 414 insertions(+), 8 deletions(-)

diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
index fa569f97ca..6cdbb0acbb 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
@@ -19,16 +19,53 @@
 
 package org.apache.texera.amber.core.storage.model
 
-import org.apache.texera.common.config.EnvironmentalVariable
+import com.sun.net.httpserver.{HttpExchange, HttpServer}
+import org.apache.texera.common.config.{EnvironmentalVariable, StorageConfig}
 import org.apache.texera.amber.core.storage.ResourceType
+import org.apache.texera.common.tags.NonParallelTest
 import org.scalatest.flatspec.AnyFlatSpec
 import org.scalatest.matchers.should.Matchers
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
 
-import java.net.{URI, URLEncoder}
+import java.io.{ByteArrayInputStream, InputStream}
+import java.net.{InetSocketAddress, URI, URLDecoder, URLEncoder}
 import java.nio.charset.StandardCharsets
-import java.nio.file.Paths
+import java.nio.file.{Files, Paths}
+import java.util.concurrent.{ConcurrentLinkedQueue, ExecutorService, Executors}
+import scala.jdk.CollectionConverters._
 
-class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
+/**
+  * Spec for [[LakeFSFileDocument]]: URI parsing, and the read path that turns the parsed
+  * repository/commit/file triple into bytes.
+  *
+  * The read path is driven against a loopback [[HttpServer]] that speaks the three HTTP hops the
+  * document can make — the lakeFS `objects/stat?presign=true` lookup, a GET of the presigned
+  * address that lookup returns, and the direct lakeFS `objects` download used as a fallback. No
+  * lakeFS server (and no Docker) is involved.
+  *
+  * Every hop is served by the same stub, and the presigned address points back at it, which is
+  * what lets a test tell "served through the presigned URL" apart from "fell back to lakeFS": the
+  * two routes return different bytes. Against a real lakeFS both paths return the same file, so
+  * the fallback could silently become the only live path.
+  *
+  * These tests exercise the `userJwtToken.isEmpty` half of `asInputStream`. That is the half every
+  * test JVM takes: `USER_JWT_TOKEN` is injected only into the pod env of a user-created computing
+  * unit (`ComputingUnitManagingResource`), never in CI or a dev shell. The non-empty half — which
+  * asks a file-service presign endpoint on a fixed port for the URL — is left to file-service's own
+  * tests; reaching it here would need a hardcoded port in `build.sbt`'s test grouping.
+  *
+  * Tagged [[NonParallelTest]] so `common/workflow-core/build.sbt` gives this suite its own forked
+  * JVM. That is load-bearing: `beforeAll` repoints `StorageConfig.lakefsEndpoint`, a JVM-wide
+  * `var` that `LakeFSStorageClient.apiClient` (a `lazy val`) captures once per JVM, and
+  * `LakeFSStorageClientSpec` / `LakeFSStorageClientMtimeSpec` point that same endpoint at their own
+  * servers.
+  */
+@NonParallelTest
+class LakeFSFileDocumentSpec
+    extends AnyFlatSpec
+    with Matchers
+    with BeforeAndAfterAll
+    with BeforeAndAfterEach {
 
   // Realistic 40-char git commit hash, mirroring the URIs produced by FileResolver
   // (format: dataset:///{repositoryName}/{versionHash}/{fileRelativePath}).
@@ -37,6 +74,129 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
   // URI parsing is shared by every resource type; exercise it through the dataset type.
   private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri, ResourceType.Dataset)
 
+  // -----------------------------------------------------------------------------------------
+  // Loopback stub
+  // -----------------------------------------------------------------------------------------
+
+  /** One recorded request. Fields are read directly; never destructured in a `case` pattern. */
+  private case class Hit(method: String, path: String, query: Map[String, String])
+
+  private val requests = new ConcurrentLinkedQueue[Hit]()
+  private var server: HttpServer = _
+  private var serverPool: ExecutorService = _
+
+  /** Request paths that should answer 500 instead of being served, for this one test. */
+  @volatile private var failing: Set[String] = Set.empty
+
+  private val repositoryName = "texera-file-doc"
+  private val fileName = "records.csv"
+  // Most dataset and model files live under a directory, so the read path is driven over both
+  // shapes: `fileName` at the repository root, and this nested path.
+  //
+  // getFileRelativePath() renders a parsed path with the *platform* separator, so every expectation
+  // over a multi-segment path — here and in the URI block above — is derived through Paths.get
+  // exactly as production renders it. Spelling out a literal "nested/dir/records.csv" instead would
+  // assert a normalization the class does not do (it would fail on Windows today), not a choice the
+  // class makes.
+  private val nestedSegments = Seq("nested", "dir", fileName)
+
+  private def statPath = s"/api/v1/repositories/$repositoryName/refs/$versionHash/objects/stat"
+  private def objectPath = s"/api/v1/repositories/$repositoryName/refs/$versionHash/objects"
+  private def presignedPath = "/signed-blob/records.csv"
+
+  // Longer than asFile's 1024-byte copy buffer and not a multiple of it (2502 bytes), so the copy
+  // loop runs several times and ends on a short read that must not be padded out to a full buffer.
+  private val presignedContent: String = "presigned-payload;" * 139
+  // Deliberately different bytes, and a different length, from the presigned payload.
+  private val fallbackContent: String = "direct-lakefs-download"
+
+  private def presignedUrl: String =
+    s"http://127.0.0.1:${server.getAddress.getPort}$presignedPath"
+
+  private def objectStatsJson: String =
+    s"""{"path":"$fileName","path_type":"object","physical_address":"$presignedUrl",
+       |"checksum":"chk","mtime":1700000000,"size_bytes":${presignedContent.length}}""".stripMargin
+
+  private def handle(exchange: HttpExchange): Unit = {
+    try {
+      val uri = exchange.getRequestURI
+      val query = Option(uri.getRawQuery)
+        .filter(_.nonEmpty)
+        .map(_.split("&").toList.map { pair =>
+          def dec(s: String) = URLDecoder.decode(s, StandardCharsets.UTF_8.name())
+          pair.indexOf('=') match {
+            case -1 => dec(pair) -> ""
+            case i  => dec(pair.substring(0, i)) -> dec(pair.substring(i + 1))
+          }
+        }.toMap)
+        .getOrElse(Map.empty[String, String])
+      requests.add(Hit(exchange.getRequestMethod, uri.getPath, query))
+
+      val (status, body) =
+        if (failing.contains(uri.getPath)) (500, """{"message":"stubbed failure"}""")
+        else
+          uri.getPath match {
+            case p if p == statPath      => (200, objectStatsJson)
+            case p if p == presignedPath => (200, presignedContent)
+            case p if p == objectPath    => (200, fallbackContent)
+            case p                       => (501, s"""{"message":"no stub route for $p"}""")
+          }
+
+      val bytes = body.getBytes(StandardCharsets.UTF_8)
+      val contentType =
+        if (uri.getPath == statPath || failing.contains(uri.getPath)) "application/json"
+        else "application/octet-stream"
+      exchange.getResponseHeaders.set("Content-Type", contentType)
+      exchange.sendResponseHeaders(status, bytes.length.toLong)
+      exchange.getResponseBody.write(bytes)
+    } finally exchange.close()
+  }
+
+  override def beforeAll(): Unit = {
+    super.beforeAll()
+    server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0)
+    server.createContext("/", (exchange: HttpExchange) => handle(exchange))
+    serverPool = Executors.newFixedThreadPool(2)
+    server.setExecutor(serverPool)
+    server.start()
+    // Must happen before anything forces LakeFSStorageClient.apiClient (a JVM-wide lazy val).
+    // Nothing above this line touches that object, and no other suite shares this forked JVM.
+    StorageConfig.lakefsEndpoint = s"http://127.0.0.1:${server.getAddress.getPort}/api/v1"
+  }
+
+  override def afterAll(): Unit = {
+    try {
+      if (server != null) server.stop(0)
+      if (serverPool != null) serverPool.shutdownNow()
+    } finally super.afterAll()
+  }
+
+  override def beforeEach(): Unit = {
+    super.beforeEach()
+    requests.clear()
+    failing = Set.empty
+  }
+
+  private def hits: List[Hit] = requests.asScala.toList
+
+  /** A document over the file the stub serves. */
+  private def servedDoc(): LakeFSFileDocument =
+    datasetDoc(new URI(s"dataset:///$repositoryName/$versionHash/$fileName"))
+
+  /** The same file two directories deep — the shape most dataset and model files have. */
+  private def nestedDoc(): LakeFSFileDocument =
+    datasetDoc(new URI(s"dataset:///$repositoryName/$versionHash/${nestedSegments.mkString("/")}"))
+
+  /** How the document renders that nested path, derived the same way production renders it. */
+  private def nestedRelativePath: String =
+    Paths.get(nestedSegments.head, nestedSegments.tail: _*).toString
+
+  private def readFully(doc: LakeFSFileDocument): String = {
+    val stream = doc.asInputStream()
+    try new String(stream.readAllBytes(), StandardCharsets.UTF_8)
+    finally stream.close()
+  }
+
   "LakeFSFileDocument" should "parse a valid 3-segment dataset URI into its components" in {
     val uri = new URI(s"dataset:///test_dataset/$versionHash/1.txt")
     val doc = datasetDoc(uri)
@@ -184,4 +344,145 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
       sys.env.getOrElse(EnvironmentalVariable.ENV_USER_JWT_TOKEN, "").trim
     LakeFSFileDocument.userJwtToken shouldBe expected
   }
+
+  // -----------------------------------------------------------------------------------------
+  // asInputStream
+  // -----------------------------------------------------------------------------------------
+
+  "asInputStream" should "read the bytes through the presigned URL, not through lakeFS" in {
+    readFully(servedDoc()) shouldBe presignedContent
+
+    // Exactly two hops, in order: the presign lookup, then the address it returned. The point of
+    // presigning is that the bytes never travel through the lakeFS server, so a third hit on the
+    // direct-download route would mean the optimization is not happening at all.
+    hits.map(_.path) shouldEqual List(statPath, presignedPath)
+    // The lookup must name this document's own repository/commit/file triple. Repository and
+    // commit are pinned by the URL path above; the file is a query parameter.
+    hits.head.query.get("path") shouldBe Some(fileName)
+  }
+
+  // Not asserted below: the WARN `fallbackToLakeFS` logs, which is the only trace an operator gets
+  // that presigning silently degraded. workflow-core's test classpath has no SLF4J provider — the
+  // run prints "No SLF4J providers were found", every logger is a NOP, and scala-logging's
+  // isWarnEnabled guard means the call is not even evaluated — so a capturing appender has nothing
+  // to attach to. Pinning it would take a logback-classic test dependency for the whole module.
+  it should "fall back to a direct lakeFS download when the presign lookup fails" in {
+    failing = Set(statPath)
+
+    // A lakeFS deployment with presigning disabled (or an object store that refuses to sign)
+    // must still yield the file rather than failing the read.
+    readFully(servedDoc()) shouldBe fallbackContent
+
+    hits.map(_.path) shouldEqual List(statPath, objectPath)
+    hits.last.query.get("path") shouldBe Some(fileName)
+  }
+
+  it should "fall back to a direct lakeFS download when the presigned URL cannot be read" in {
+    failing = Set(presignedPath)
+
+    // Distinct from the case above: here the lookup succeeded and the *address* is unusable — an
+    // expired signature, or an object store the JVM cannot reach. The fallback covers both, so a
+    // stale presigned URL is not a read failure.
+    readFully(servedDoc()) shouldBe fallbackContent
+
+    hits.map(_.path) shouldEqual List(statPath, presignedPath, objectPath)
+  }
+
+  it should "name the whole nested relative path on both hops, not the file's last segment" in {
+    failing = Set(statPath)
+
+    // A nested file driven through the fallback puts the relative path on both hops of one read —
+    // the presign lookup's query, then the direct download's — so this pins both places the read
+    // path passes the file. Asking for "records.csv" alone would address a different object, one at
+    // the repository root, which for most files does not exist: a silent 404 on both hops.
+    readFully(nestedDoc()) shouldBe fallbackContent
+
+    hits.map(_.path) shouldEqual List(statPath, objectPath)
+    hits.map(_.query.get("path")) shouldEqual
+      List(Some(nestedRelativePath), Some(nestedRelativePath))
+  }
+
+  // -----------------------------------------------------------------------------------------
+  // asFile / clear
+  // -----------------------------------------------------------------------------------------
+
+  "asFile" should "copy the fetched bytes to a temp file and reuse that file on later calls" in {
+    val doc = servedDoc()
+    val first = doc.asFile()
+    try {
+      // The payload is longer than the 1024-byte copy buffer and not a multiple of it, so this
+      // pins the buffered copy loop: writing a whole buffer on the final short read would append
+      // trailing garbage, and stopping after one buffer would truncate.
+      new String(Files.readAllBytes(first.toPath), StandardCharsets.UTF_8) shouldBe presignedContent
+
+      // Callers hand the File to operator code that opens it more than once, so the document must
+      // memoize it: re-materializing would download the file again and leak the earlier temp file.
+      requests.clear()
+      assert(doc.asFile() eq first)
+      // Same File *and* nothing on the wire: re-fetching into the memoized path would keep the
+      // identity assertion above green while re-downloading the whole object on every call.
+      hits shouldBe empty
+    } finally Files.deleteIfExists(first.toPath)
+  }
+
+  it should "close the stream it copied from" in {
+    // A leaked stream here is a leaked socket per versioned-file read in a long-running worker
+    // (the shape of the Iceberg reader leak, #6882). The HTTP stub cannot observe the close, so
+    // hand asFile a stream that records it: asInputStream is overridable and asFile calls it
+    // virtually, which is exactly the seam production uses.
+    var closes = 0
+    val doc = new LakeFSFileDocument(
+      new URI(s"dataset:///$repositoryName/$versionHash/$fileName"),
+      ResourceType.Dataset
+    ) {
+      override def asInputStream(): InputStream =
+        new ByteArrayInputStream(presignedContent.getBytes(StandardCharsets.UTF_8)) {
+          override def close(): Unit = {
+            closes += 1
+            super.close()
+          }
+        }
+    }
+
+    val file = doc.asFile()
+    try {
+      new String(Files.readAllBytes(file.toPath), StandardCharsets.UTF_8) shouldBe presignedContent
+      closes shouldBe 1
+    } finally Files.deleteIfExists(file.toPath)
+  }
+
+  "clear" should "delete the materialized temp file and let a later asFile re-materialize it" in {
+    val doc = servedDoc()
+    val first = doc.asFile()
+    // A *materialized* file, not merely the empty placeholder Files.createTempFile leaves behind:
+    // existence alone is the JDK's guarantee, and the precondition this test destroys and rebuilds
+    // is that the download landed on disk.
+    Files.size(first.toPath) shouldBe presignedContent.length.toLong
+
+    doc.clear()
+    // The temp file lives in the OS temp directory for the life of the JVM otherwise; leaking one
+    // per read is how a long-running worker fills its disk.
+    Files.exists(first.toPath) shouldBe false
+
+    // clear also drops the memo, so the document stays usable instead of handing back a File that
+    // no longer exists.
+    val second = doc.asFile()
+    try {
+      second.getPath should not be first.getPath
+      new String(Files.readAllBytes(second.toPath), StandardCharsets.UTF_8) shouldBe
+        presignedContent
+    } finally Files.deleteIfExists(second.toPath)
+  }
+
+  it should "do nothing when no temp file has been materialized" in {
+    val doc = servedDoc()
+    // clear() is called on every document at the end of a workflow, materialized or not.
+    noException should be thrownBy doc.clear()
+
+    doc.asFile()
+    doc.clear()
+    // A second clear must not retry the delete on the path the first one removed, which would
+    // throw NoSuchFileException.
+    noException should be thrownBy doc.clear()
+  }
 }
diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala
index fbd4289c97..41cb7b0e80 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala
@@ -48,7 +48,11 @@ private object LakeFSStubServer {
       body: String
   )
 
-  final case class StubResponse(status: Int, body: String = "")
+  final case class StubResponse(
+      status: Int,
+      body: String = "",
+      headers: Map[String, String] = Map.empty
+  )
 }
 
 /**
@@ -95,7 +99,13 @@ class LakeFSStorageClientSpec
 
   import LakeFSStubServer._
 
-  private val requests = new ConcurrentLinkedQueue[StubRequest]()
+  /**
+    * Recorded requests, each paired with its request headers (keys lower-cased, since
+    * [[HttpExchange]] normalizes header capitalization). The headers ride alongside rather than
+    * inside [[StubRequest]] so that the two dozen `case StubRequest(method, path, query, body)`
+    * routes below keep their arity — only the `put` test reads them.
+    */
+  private val requests = new ConcurrentLinkedQueue[(StubRequest, Map[String, String])]()
 
   private val notStubbed: StubRequest => StubResponse =
     req => StubResponse(501, s"""{"message":"no stub route for ${req.method} ${req.path}"}""")
@@ -120,14 +130,22 @@ class LakeFSStorageClientSpec
         }.toMap)
         .getOrElse(Map.empty[String, String])
 
+      val headers = exchange.getRequestHeaders.asScala.map {
+        case (name, values) => name.toLowerCase -> values.asScala.mkString(",")
+      }.toMap
+
       val request =
         StubRequest(exchange.getRequestMethod, exchange.getRequestURI.getPath, query, body)
-      requests.add(request)
+      requests.add((request, headers))
 
       val response =
         try route(request)
         catch { case t: Throwable => StubResponse(500, s"""{"message":"${t.getClass.getName}"}""") }
 
+      response.headers.foreach {
+        case (name, value) => exchange.getResponseHeaders.set(name, value)
+      }
+
       val bytes = response.body.getBytes(UTF_8)
       if (bytes.isEmpty) {
         // -1 means "no response body"; the SDK maps 204 to a null (Unit) return.
@@ -169,7 +187,10 @@ class LakeFSStorageClientSpec
   private def stub(routes: PartialFunction[StubRequest, StubResponse]): Unit =
     route = req => routes.applyOrElse(req, notStubbed)
 
-  private def recorded: List[StubRequest] = requests.asScala.toList
+  private def recorded: List[StubRequest] = requests.asScala.map(_._1).toList
+
+  /** Origin of the stub, for the calls that take a full URL rather than going through the SDK. */
+  private def stubBaseUrl: String = s"http://127.0.0.1:${server.getAddress.getPort}"
 
   /**
     * True only for the FIRST request this test makes to `path`. Page-1 routes are guarded with it so
@@ -185,6 +206,12 @@ class LakeFSStorageClientSpec
     recorded.head
   }
 
+  /** Headers of the single recorded request, keys lower-cased. */
+  private def onlyRequestHeaders: Map[String, String] = {
+    recorded should have size 1
+    requests.asScala.head._2
+  }
+
   private val mapper = new ObjectMapper()
   private def json(body: String) = mapper.readTree(body)
 
@@ -554,6 +581,84 @@ class LakeFSStorageClientSpec
     onlyRequest.query.get("path") shouldBe Some(path)
   }
 
+  // `put` is the middle step of the presigned lifecycle. It bypasses the lakeFS SDK entirely and
+  // talks to the object store over a raw HttpURLConnection, so the stub stands in for the store.
+
+  "put" should "upload exactly the first `len` bytes and return the ETag with quotes stripped" in {
+    val partPath = "/presigned/part-1"
+    stub {
+      case StubRequest("PUT", p, _, _) if p == partPath =>
+        // Object stores return the ETag quoted; callers feed it straight into the completion
+        // body, where lakeFS rejects a quoted value.
+        StubResponse(200, headers = Map("ETag" -> "\"d41d8cd98f00b204e9800998ecf8427e\""))
+    }
+
+    // A buffer longer than the part: the last part of a multipart upload is a partially-filled
+    // read buffer, and uploading its stale tail corrupts the assembled object.
+    val etag = LakeFSStorageClient.put("abcdefgh".getBytes(UTF_8), 5, s"$stubBaseUrl$partPath", 1)
+
+    etag shouldEqual "d41d8cd98f00b204e9800998ecf8427e"
+    onlyRequest.method shouldEqual "PUT"
+    onlyRequest.body shouldEqual "abcde"
+    // The part must be framed with a Content-Length, not chunked: a presigned S3/GCS PUT is signed
+    // for a known length and answers `Transfer-Encoding: chunked` with SignatureDoesNotMatch or
+    // 501, while a loopback stub de-chunks the body and notices nothing.
+    //
+    // This pins the *framing*, not the streaming: the JDK's default buffered mode also computes a
+    // Content-Length for a body this small, so dropping the streaming-mode call altogether is
+    // indistinguishable over a stub. Only the real gain of streaming — not holding the part in the
+    // heap — is unobservable here, and that is a memory property, not a wire one.
+    onlyRequestHeaders.get("content-length") shouldBe Some("5")
+    onlyRequestHeaders.get("transfer-encoding") shouldBe None
+  }
+
+  it should "treat both 200 and 201 as a successful part upload" in {
+    // S3 answers a part PUT with 200, other object stores with 201. Accepting only one would
+    // abort an otherwise-complete multipart upload against half the supported backends.
+    Seq(200 -> "etag-from-200", 201 -> "etag-from-201").foreach {
+      case (status, etag) =>
+        requests.clear()
+        val partPath = s"/presigned/part-ok-$status"
+        stub {
+          case StubRequest("PUT", p, _, _) if p == partPath =>
+            StubResponse(status, headers = Map("ETag" -> etag))
+        }
+
+        withClue(s"HTTP $status: ") {
+          LakeFSStorageClient.put(
+            "x".getBytes(UTF_8),
+            1,
+            s"$stubBaseUrl$partPath",
+            2
+          ) shouldEqual etag
+        }
+    }
+  }
+
+  it should "name the part and the status when the object store rejects the upload" in {
+    // 403 is the live failure: an expired or mis-signed URL. 204 and 307 are the shape check — the
+    // guard has to stay "exactly 200 or 201" rather than widen to "4xx and worse", because neither
+    // of those two carries an ETag: accepting one books a part that was never stored, and the
+    // completion call then either NPEs on the missing ETag or assembles an object with a hole in it.
+    // 307 in particular is what an S3 region/endpoint mismatch answers.
+    Seq(403, 204, 307).foreach { status =>
+      requests.clear()
+      val partPath = s"/presigned/part-$status"
+      stub {
+        case StubRequest("PUT", p, _, _) if p == partPath => StubResponse(status)
+      }
+
+      val ex = intercept[RuntimeException] {
+        LakeFSStorageClient.put("x".getBytes(UTF_8), 1, s"$stubBaseUrl$partPath", 4)
+      }
+      // A multipart upload fails one part at a time; without the part number and the status in the
+      // message there is nothing in the log to say which part failed or whether it is retryable.
+      withClue(s"HTTP $status: ") {
+        ex.getMessage shouldEqual s"Part 4 upload failed (HTTP $status)"
+      }
+    }
+  }
+
   // ---------------------------------------------------------------------------------------------
   // Branch and object mutations
   // ---------------------------------------------------------------------------------------------