You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/22 10:53:28 UTC
(texera) 01/01: test(workflow-core): cover the lakeFS file document against the loopback stub (#7831)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7831-fdc919110b15d358055b11cb35bc1ea4ee006dee
in repository https://gitbox.apache.org/repos/asf/texera.git
commit b7c33b0892db49a94a3b15c81eed4ebcbc12882d
Author: Xinyuan Lin <xi...@uci.edu>
AuthorDate: Sat Aug 22 10:53:06 2026 +0000
test(workflow-core): cover the lakeFS file document against the loopback stub (#7831)
### What changes were proposed in this PR?
Test-only. **No production file changes** — `git diff -- '*/src/main/*'`
is empty.
`LakeFSFileDocument` looked Docker-gated and is not:
`LakeFSStorageClientSpec` (merged in #7273) already drives
`LakeFSStorageClient` against an in-process `com.sun.net.httpserver`
loopback stub, and already covers the exact two calls the document
makes. This PR reuses that harness.
Measured with `WorkflowCore/jacoco` under an identical filter for the
before and after runs (the Docker-backed `LakeFSStorageClientMtimeSpec`
is excluded from **both**, so these differ from the Codecov figures,
which include it):
| File | Lines | Branches |
|---|---|---|
| `LakeFSFileDocument.scala` | 32/80 (40.0%) → **65/80 (81.2%)** | 16/46
→ **27/46** |
| `LakeFSStorageClient.scala` | 120/146 (82.2%) → **133/146 (91.1%)** |
41/60 → **45/60** |
**+33 lines and +11 branch arms** on the document, **+13 lines** on the
client. Tests **38 → 49**.
Covered: `asInputStream`'s presign happy path and its fallback (the
local `def fallbackToLakeFS` lifts to `fallbackToLakeFS$1`, a counted
method rather than an `$anonfun$`, which is why these lines move the
number at all), `asFile` both arms including the `tempFile` memoization,
`clear` both arms, and `LakeFSStorageClient.put`.
The 15 lines still missed on the document are exactly the out-of-scope
region — lines 133-161, the `userJwtToken`-non-empty file-service
presign branch. Reaching it needs a forked JVM with `USER_JWT_TOKEN` set
**and** a presign endpoint on a *fixed* port known before JVM start,
i.e. a `build.sbt` `testGrouping` change plus a hardcodable port that
can collide in CI. Deliberately excluded.
### Verification
Review proposed 8 findings. **Six were real survivors and are now
killed**, each proved red on a named test after the repair; two are
genuinely unpinnable here and are recorded in the spec rather than
papered over. Suite 47 → 49 after repair.
| Survivor found by review | How it is now killed |
|---|---|
| both `asInputStream` call sites reduced to
`fileRelativePath.getFileName.toString` | a nested-fixture read test
that kills the pair *and* each site alone |
| widening the `put` success guard to `code >= 400` | rejection test
extended to 403/204/307; the 204 and 307 single-leg widenings each die
separately |
| deleting `inputStream.close()` | a recording-subclass test,
deterministic on any platform |
| re-downloading into the memoized `File` | `requests.clear()` plus
`hits shouldBe empty` |
| dropping `setChunkedStreamingMode` | the stub now records request
headers; the put test pins `content-length: 5` and the absence of
`transfer-encoding` |
| `Files.exists shouldBe true` was JDK-guaranteed | now `Files.size ==
2502` — and the mutant fails *at that line* rather than 12 lines later |
**Two limits, argued rather than cargo-culted.** The fallback
`logger.warn` at :110 cannot be pinned: `workflow-core`'s test classpath
has no SLF4J provider at all ("No SLF4J providers were found", NOP
loggers), and scala-logging's `isWarnEnabled` guard means the call is
not even evaluated — a capturing appender has nothing to attach to. And
`deleteRepo` (1 line) was left alone: it is
`repoApi.deleteRepository(name).execute()` with no branch, so the only
assertable fact is the generated SDK's own URL template. The existing
spec already documents that rationale and I did not contradict it for
one line.
### Two production bugs found while doing this, reported not fixed
Both are out of scope for a test-only PR and neither is cemented by any
new test.
1. **`LakeFSFileDocument.scala:204` renders the relative path with the
platform separator.** On a Windows worker, every nested dataset/model
file is requested as `dir\file.csv` in the lakeFS `path` query parameter
— 404, silent fallback, 404 again. A reviewer's suggested assertion of
`Some("nested/dir/records.csv")` would be red against today's
production, which is how this surfaced. Fix is a one-line `'/'`-join and
wants its own PR.
2. **`tempFileStream.close()` is load-bearing on Windows only.**
Deleting it fails 4 tests here with "file is being used by another
process" (the cleanup hits an open write handle) while shipping green
through Linux CI. `FileOutputStream` is unbuffered, so no portable
in-process assertion can observe it; wrapping the copy in
`try`/`finally` is production work.
### Any related issues, documentation, discussions?
Closes #7830
### How was this PR tested?
```
sbt "WorkflowCore/testOnly org.apache.texera.amber.core.storage.model.LakeFSFileDocumentSpec org.apache.texera.amber.core.storage.util.LakeFSStorageClientSpec"
```
```
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 49, failed 0, canceled 0, ignored 0, pending 0
```
`scalafmtCheckAll` and `scalafixAll --check` both pass.
`LakeFSFileDocumentSpec` gains `@NonParallelTest` so it gets its own
forked JVM, matching the sibling spec.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
---------
Signed-off-by: Xinyuan Lin <xi...@uci.edu>
Co-authored-by: Copilot Autofix powered by AI <17...@users.noreply.github.com>
---
.../storage/model/LakeFSFileDocumentSpec.scala | 309 ++++++++++++++++++++-
.../storage/util/LakeFSStorageClientSpec.scala | 113 +++++++-
2 files changed, 414 insertions(+), 8 deletions(-)
diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
index fa569f97ca..6cdbb0acbb 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
@@ -19,16 +19,53 @@
package org.apache.texera.amber.core.storage.model
-import org.apache.texera.common.config.EnvironmentalVariable
+import com.sun.net.httpserver.{HttpExchange, HttpServer}
+import org.apache.texera.common.config.{EnvironmentalVariable, StorageConfig}
import org.apache.texera.amber.core.storage.ResourceType
+import org.apache.texera.common.tags.NonParallelTest
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
-import java.net.{URI, URLEncoder}
+import java.io.{ByteArrayInputStream, InputStream}
+import java.net.{InetSocketAddress, URI, URLDecoder, URLEncoder}
import java.nio.charset.StandardCharsets
-import java.nio.file.Paths
+import java.nio.file.{Files, Paths}
+import java.util.concurrent.{ConcurrentLinkedQueue, ExecutorService, Executors}
+import scala.jdk.CollectionConverters._
-class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
+/**
+ * Spec for [[LakeFSFileDocument]]: URI parsing, and the read path that turns the parsed
+ * repository/commit/file triple into bytes.
+ *
+ * The read path is driven against a loopback [[HttpServer]] that speaks the three HTTP hops the
+ * document can make — the lakeFS `objects/stat?presign=true` lookup, a GET of the presigned
+ * address that lookup returns, and the direct lakeFS `objects` download used as a fallback. No
+ * lakeFS server (and no Docker) is involved.
+ *
+ * Every hop is served by the same stub, and the presigned address points back at it, which is
+ * what lets a test tell "served through the presigned URL" apart from "fell back to lakeFS": the
+ * two routes return different bytes. Against a real lakeFS both paths return the same file, so
+ * the fallback could silently become the only live path.
+ *
+ * These tests exercise the `userJwtToken.isEmpty` half of `asInputStream`. That is the half every
+ * test JVM takes: `USER_JWT_TOKEN` is injected only into the pod env of a user-created computing
+ * unit (`ComputingUnitManagingResource`), never in CI or a dev shell. The non-empty half — which
+ * asks a file-service presign endpoint on a fixed port for the URL — is left to file-service's own
+ * tests; reaching it here would need a hardcoded port in `build.sbt`'s test grouping.
+ *
+ * Tagged [[NonParallelTest]] so `common/workflow-core/build.sbt` gives this suite its own forked
+ * JVM. That is load-bearing: `beforeAll` repoints `StorageConfig.lakefsEndpoint`, a JVM-wide
+ * `var` that `LakeFSStorageClient.apiClient` (a `lazy val`) captures once per JVM, and
+ * `LakeFSStorageClientSpec` / `LakeFSStorageClientMtimeSpec` point that same endpoint at their own
+ * servers.
+ */
+@NonParallelTest
+class LakeFSFileDocumentSpec
+ extends AnyFlatSpec
+ with Matchers
+ with BeforeAndAfterAll
+ with BeforeAndAfterEach {
// Realistic 40-char git commit hash, mirroring the URIs produced by FileResolver
// (format: dataset:///{repositoryName}/{versionHash}/{fileRelativePath}).
@@ -37,6 +74,129 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
// URI parsing is shared by every resource type; exercise it through the dataset type.
private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri, ResourceType.Dataset)
+ // -----------------------------------------------------------------------------------------
+ // Loopback stub
+ // -----------------------------------------------------------------------------------------
+
+ /** One recorded request. Fields are read directly; never destructured in a `case` pattern. */
+ private case class Hit(method: String, path: String, query: Map[String, String])
+
+ private val requests = new ConcurrentLinkedQueue[Hit]()
+ private var server: HttpServer = _
+ private var serverPool: ExecutorService = _
+
+ /** Request paths that should answer 500 instead of being served, for this one test. */
+ @volatile private var failing: Set[String] = Set.empty
+
+ private val repositoryName = "texera-file-doc"
+ private val fileName = "records.csv"
+ // Most dataset and model files live under a directory, so the read path is driven over both
+ // shapes: `fileName` at the repository root, and this nested path.
+ //
+ // getFileRelativePath() renders a parsed path with the *platform* separator, so every expectation
+ // over a multi-segment path — here and in the URI block above — is derived through Paths.get
+ // exactly as production renders it. Spelling out a literal "nested/dir/records.csv" instead would
+ // assert a normalization the class does not do (it would fail on Windows today), not a choice the
+ // class makes.
+ private val nestedSegments = Seq("nested", "dir", fileName)
+
+ private def statPath = s"/api/v1/repositories/$repositoryName/refs/$versionHash/objects/stat"
+ private def objectPath = s"/api/v1/repositories/$repositoryName/refs/$versionHash/objects"
+ private def presignedPath = "/signed-blob/records.csv"
+
+ // Longer than asFile's 1024-byte copy buffer and not a multiple of it (2502 bytes), so the copy
+ // loop runs several times and ends on a short read that must not be padded out to a full buffer.
+ private val presignedContent: String = "presigned-payload;" * 139
+ // Deliberately different bytes, and a different length, from the presigned payload.
+ private val fallbackContent: String = "direct-lakefs-download"
+
+ private def presignedUrl: String =
+ s"http://127.0.0.1:${server.getAddress.getPort}$presignedPath"
+
+ private def objectStatsJson: String =
+ s"""{"path":"$fileName","path_type":"object","physical_address":"$presignedUrl",
+ |"checksum":"chk","mtime":1700000000,"size_bytes":${presignedContent.length}}""".stripMargin
+
+ private def handle(exchange: HttpExchange): Unit = {
+ try {
+ val uri = exchange.getRequestURI
+ val query = Option(uri.getRawQuery)
+ .filter(_.nonEmpty)
+ .map(_.split("&").toList.map { pair =>
+ def dec(s: String) = URLDecoder.decode(s, StandardCharsets.UTF_8.name())
+ pair.indexOf('=') match {
+ case -1 => dec(pair) -> ""
+ case i => dec(pair.substring(0, i)) -> dec(pair.substring(i + 1))
+ }
+ }.toMap)
+ .getOrElse(Map.empty[String, String])
+ requests.add(Hit(exchange.getRequestMethod, uri.getPath, query))
+
+ val (status, body) =
+ if (failing.contains(uri.getPath)) (500, """{"message":"stubbed failure"}""")
+ else
+ uri.getPath match {
+ case p if p == statPath => (200, objectStatsJson)
+ case p if p == presignedPath => (200, presignedContent)
+ case p if p == objectPath => (200, fallbackContent)
+ case p => (501, s"""{"message":"no stub route for $p"}""")
+ }
+
+ val bytes = body.getBytes(StandardCharsets.UTF_8)
+ val contentType =
+ if (uri.getPath == statPath || failing.contains(uri.getPath)) "application/json"
+ else "application/octet-stream"
+ exchange.getResponseHeaders.set("Content-Type", contentType)
+ exchange.sendResponseHeaders(status, bytes.length.toLong)
+ exchange.getResponseBody.write(bytes)
+ } finally exchange.close()
+ }
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0)
+ server.createContext("/", (exchange: HttpExchange) => handle(exchange))
+ serverPool = Executors.newFixedThreadPool(2)
+ server.setExecutor(serverPool)
+ server.start()
+ // Must happen before anything forces LakeFSStorageClient.apiClient (a JVM-wide lazy val).
+ // Nothing above this line touches that object, and no other suite shares this forked JVM.
+ StorageConfig.lakefsEndpoint = s"http://127.0.0.1:${server.getAddress.getPort}/api/v1"
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ if (server != null) server.stop(0)
+ if (serverPool != null) serverPool.shutdownNow()
+ } finally super.afterAll()
+ }
+
+ override def beforeEach(): Unit = {
+ super.beforeEach()
+ requests.clear()
+ failing = Set.empty
+ }
+
+ private def hits: List[Hit] = requests.asScala.toList
+
+ /** A document over the file the stub serves. */
+ private def servedDoc(): LakeFSFileDocument =
+ datasetDoc(new URI(s"dataset:///$repositoryName/$versionHash/$fileName"))
+
+ /** The same file two directories deep — the shape most dataset and model files have. */
+ private def nestedDoc(): LakeFSFileDocument =
+ datasetDoc(new URI(s"dataset:///$repositoryName/$versionHash/${nestedSegments.mkString("/")}"))
+
+ /** How the document renders that nested path, derived the same way production renders it. */
+ private def nestedRelativePath: String =
+ Paths.get(nestedSegments.head, nestedSegments.tail: _*).toString
+
+ private def readFully(doc: LakeFSFileDocument): String = {
+ val stream = doc.asInputStream()
+ try new String(stream.readAllBytes(), StandardCharsets.UTF_8)
+ finally stream.close()
+ }
+
"LakeFSFileDocument" should "parse a valid 3-segment dataset URI into its components" in {
val uri = new URI(s"dataset:///test_dataset/$versionHash/1.txt")
val doc = datasetDoc(uri)
@@ -184,4 +344,145 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with Matchers {
sys.env.getOrElse(EnvironmentalVariable.ENV_USER_JWT_TOKEN, "").trim
LakeFSFileDocument.userJwtToken shouldBe expected
}
+
+ // -----------------------------------------------------------------------------------------
+ // asInputStream
+ // -----------------------------------------------------------------------------------------
+
+ "asInputStream" should "read the bytes through the presigned URL, not through lakeFS" in {
+ readFully(servedDoc()) shouldBe presignedContent
+
+ // Exactly two hops, in order: the presign lookup, then the address it returned. The point of
+ // presigning is that the bytes never travel through the lakeFS server, so a third hit on the
+ // direct-download route would mean the optimization is not happening at all.
+ hits.map(_.path) shouldEqual List(statPath, presignedPath)
+ // The lookup must name this document's own repository/commit/file triple. Repository and
+ // commit are pinned by the URL path above; the file is a query parameter.
+ hits.head.query.get("path") shouldBe Some(fileName)
+ }
+
+ // Not asserted below: the WARN `fallbackToLakeFS` logs, which is the only trace an operator gets
+ // that presigning silently degraded. workflow-core's test classpath has no SLF4J provider — the
+ // run prints "No SLF4J providers were found", every logger is a NOP, and scala-logging's
+ // isWarnEnabled guard means the call is not even evaluated — so a capturing appender has nothing
+ // to attach to. Pinning it would take a logback-classic test dependency for the whole module.
+ it should "fall back to a direct lakeFS download when the presign lookup fails" in {
+ failing = Set(statPath)
+
+ // A lakeFS deployment with presigning disabled (or an object store that refuses to sign)
+ // must still yield the file rather than failing the read.
+ readFully(servedDoc()) shouldBe fallbackContent
+
+ hits.map(_.path) shouldEqual List(statPath, objectPath)
+ hits.last.query.get("path") shouldBe Some(fileName)
+ }
+
+ it should "fall back to a direct lakeFS download when the presigned URL cannot be read" in {
+ failing = Set(presignedPath)
+
+ // Distinct from the case above: here the lookup succeeded and the *address* is unusable — an
+ // expired signature, or an object store the JVM cannot reach. The fallback covers both, so a
+ // stale presigned URL is not a read failure.
+ readFully(servedDoc()) shouldBe fallbackContent
+
+ hits.map(_.path) shouldEqual List(statPath, presignedPath, objectPath)
+ }
+
+ it should "name the whole nested relative path on both hops, not the file's last segment" in {
+ failing = Set(statPath)
+
+ // A nested file driven through the fallback puts the relative path on both hops of one read —
+ // the presign lookup's query, then the direct download's — so this pins both places the read
+ // path passes the file. Asking for "records.csv" alone would address a different object, one at
+ // the repository root, which for most files does not exist: a silent 404 on both hops.
+ readFully(nestedDoc()) shouldBe fallbackContent
+
+ hits.map(_.path) shouldEqual List(statPath, objectPath)
+ hits.map(_.query.get("path")) shouldEqual
+ List(Some(nestedRelativePath), Some(nestedRelativePath))
+ }
+
+ // -----------------------------------------------------------------------------------------
+ // asFile / clear
+ // -----------------------------------------------------------------------------------------
+
+ "asFile" should "copy the fetched bytes to a temp file and reuse that file on later calls" in {
+ val doc = servedDoc()
+ val first = doc.asFile()
+ try {
+ // The payload is longer than the 1024-byte copy buffer and not a multiple of it, so this
+ // pins the buffered copy loop: writing a whole buffer on the final short read would append
+ // trailing garbage, and stopping after one buffer would truncate.
+ new String(Files.readAllBytes(first.toPath), StandardCharsets.UTF_8) shouldBe presignedContent
+
+ // Callers hand the File to operator code that opens it more than once, so the document must
+ // memoize it: re-materializing would download the file again and leak the earlier temp file.
+ requests.clear()
+ assert(doc.asFile() eq first)
+ // Same File *and* nothing on the wire: re-fetching into the memoized path would keep the
+ // identity assertion above green while re-downloading the whole object on every call.
+ hits shouldBe empty
+ } finally Files.deleteIfExists(first.toPath)
+ }
+
+ it should "close the stream it copied from" in {
+ // A leaked stream here is a leaked socket per versioned-file read in a long-running worker
+ // (the shape of the Iceberg reader leak, #6882). The HTTP stub cannot observe the close, so
+ // hand asFile a stream that records it: asInputStream is overridable and asFile calls it
+ // virtually, which is exactly the seam production uses.
+ var closes = 0
+ val doc = new LakeFSFileDocument(
+ new URI(s"dataset:///$repositoryName/$versionHash/$fileName"),
+ ResourceType.Dataset
+ ) {
+ override def asInputStream(): InputStream =
+ new ByteArrayInputStream(presignedContent.getBytes(StandardCharsets.UTF_8)) {
+ override def close(): Unit = {
+ closes += 1
+ super.close()
+ }
+ }
+ }
+
+ val file = doc.asFile()
+ try {
+ new String(Files.readAllBytes(file.toPath), StandardCharsets.UTF_8) shouldBe presignedContent
+ closes shouldBe 1
+ } finally Files.deleteIfExists(file.toPath)
+ }
+
+ "clear" should "delete the materialized temp file and let a later asFile re-materialize it" in {
+ val doc = servedDoc()
+ val first = doc.asFile()
+ // A *materialized* file, not merely the empty placeholder Files.createTempFile leaves behind:
+ // existence alone is the JDK's guarantee, and the precondition this test destroys and rebuilds
+ // is that the download landed on disk.
+ Files.size(first.toPath) shouldBe presignedContent.length.toLong
+
+ doc.clear()
+ // The temp file lives in the OS temp directory for the life of the JVM otherwise; leaking one
+ // per read is how a long-running worker fills its disk.
+ Files.exists(first.toPath) shouldBe false
+
+ // clear also drops the memo, so the document stays usable instead of handing back a File that
+ // no longer exists.
+ val second = doc.asFile()
+ try {
+ second.getPath should not be first.getPath
+ new String(Files.readAllBytes(second.toPath), StandardCharsets.UTF_8) shouldBe
+ presignedContent
+ } finally Files.deleteIfExists(second.toPath)
+ }
+
+ it should "do nothing when no temp file has been materialized" in {
+ val doc = servedDoc()
+ // clear() is called on every document at the end of a workflow, materialized or not.
+ noException should be thrownBy doc.clear()
+
+ doc.asFile()
+ doc.clear()
+ // A second clear must not retry the delete on the path the first one removed, which would
+ // throw NoSuchFileException.
+ noException should be thrownBy doc.clear()
+ }
}
diff --git a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala
index fbd4289c97..41cb7b0e80 100644
--- a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala
+++ b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/LakeFSStorageClientSpec.scala
@@ -48,7 +48,11 @@ private object LakeFSStubServer {
body: String
)
- final case class StubResponse(status: Int, body: String = "")
+ final case class StubResponse(
+ status: Int,
+ body: String = "",
+ headers: Map[String, String] = Map.empty
+ )
}
/**
@@ -95,7 +99,13 @@ class LakeFSStorageClientSpec
import LakeFSStubServer._
- private val requests = new ConcurrentLinkedQueue[StubRequest]()
+ /**
+ * Recorded requests, each paired with its request headers (keys lower-cased, since
+ * [[HttpExchange]] normalizes header capitalization). The headers ride alongside rather than
+ * inside [[StubRequest]] so that the two dozen `case StubRequest(method, path, query, body)`
+ * routes below keep their arity — only the `put` test reads them.
+ */
+ private val requests = new ConcurrentLinkedQueue[(StubRequest, Map[String, String])]()
private val notStubbed: StubRequest => StubResponse =
req => StubResponse(501, s"""{"message":"no stub route for ${req.method} ${req.path}"}""")
@@ -120,14 +130,22 @@ class LakeFSStorageClientSpec
}.toMap)
.getOrElse(Map.empty[String, String])
+ val headers = exchange.getRequestHeaders.asScala.map {
+ case (name, values) => name.toLowerCase -> values.asScala.mkString(",")
+ }.toMap
+
val request =
StubRequest(exchange.getRequestMethod, exchange.getRequestURI.getPath, query, body)
- requests.add(request)
+ requests.add((request, headers))
val response =
try route(request)
catch { case t: Throwable => StubResponse(500, s"""{"message":"${t.getClass.getName}"}""") }
+ response.headers.foreach {
+ case (name, value) => exchange.getResponseHeaders.set(name, value)
+ }
+
val bytes = response.body.getBytes(UTF_8)
if (bytes.isEmpty) {
// -1 means "no response body"; the SDK maps 204 to a null (Unit) return.
@@ -169,7 +187,10 @@ class LakeFSStorageClientSpec
private def stub(routes: PartialFunction[StubRequest, StubResponse]): Unit =
route = req => routes.applyOrElse(req, notStubbed)
- private def recorded: List[StubRequest] = requests.asScala.toList
+ private def recorded: List[StubRequest] = requests.asScala.map(_._1).toList
+
+ /** Origin of the stub, for the calls that take a full URL rather than going through the SDK. */
+ private def stubBaseUrl: String = s"http://127.0.0.1:${server.getAddress.getPort}"
/**
* True only for the FIRST request this test makes to `path`. Page-1 routes are guarded with it so
@@ -185,6 +206,12 @@ class LakeFSStorageClientSpec
recorded.head
}
+ /** Headers of the single recorded request, keys lower-cased. */
+ private def onlyRequestHeaders: Map[String, String] = {
+ recorded should have size 1
+ requests.asScala.head._2
+ }
+
private val mapper = new ObjectMapper()
private def json(body: String) = mapper.readTree(body)
@@ -554,6 +581,84 @@ class LakeFSStorageClientSpec
onlyRequest.query.get("path") shouldBe Some(path)
}
+ // `put` is the middle step of the presigned lifecycle. It bypasses the lakeFS SDK entirely and
+ // talks to the object store over a raw HttpURLConnection, so the stub stands in for the store.
+
+ "put" should "upload exactly the first `len` bytes and return the ETag with quotes stripped" in {
+ val partPath = "/presigned/part-1"
+ stub {
+ case StubRequest("PUT", p, _, _) if p == partPath =>
+ // Object stores return the ETag quoted; callers feed it straight into the completion
+ // body, where lakeFS rejects a quoted value.
+ StubResponse(200, headers = Map("ETag" -> "\"d41d8cd98f00b204e9800998ecf8427e\""))
+ }
+
+ // A buffer longer than the part: the last part of a multipart upload is a partially-filled
+ // read buffer, and uploading its stale tail corrupts the assembled object.
+ val etag = LakeFSStorageClient.put("abcdefgh".getBytes(UTF_8), 5, s"$stubBaseUrl$partPath", 1)
+
+ etag shouldEqual "d41d8cd98f00b204e9800998ecf8427e"
+ onlyRequest.method shouldEqual "PUT"
+ onlyRequest.body shouldEqual "abcde"
+ // The part must be framed with a Content-Length, not chunked: a presigned S3/GCS PUT is signed
+ // for a known length and answers `Transfer-Encoding: chunked` with SignatureDoesNotMatch or
+ // 501, while a loopback stub de-chunks the body and notices nothing.
+ //
+ // This pins the *framing*, not the streaming: the JDK's default buffered mode also computes a
+ // Content-Length for a body this small, so dropping the streaming-mode call altogether is
+ // indistinguishable over a stub. Only the real gain of streaming — not holding the part in the
+ // heap — is unobservable here, and that is a memory property, not a wire one.
+ onlyRequestHeaders.get("content-length") shouldBe Some("5")
+ onlyRequestHeaders.get("transfer-encoding") shouldBe None
+ }
+
+ it should "treat both 200 and 201 as a successful part upload" in {
+ // S3 answers a part PUT with 200, other object stores with 201. Accepting only one would
+ // abort an otherwise-complete multipart upload against half the supported backends.
+ Seq(200 -> "etag-from-200", 201 -> "etag-from-201").foreach {
+ case (status, etag) =>
+ requests.clear()
+ val partPath = s"/presigned/part-ok-$status"
+ stub {
+ case StubRequest("PUT", p, _, _) if p == partPath =>
+ StubResponse(status, headers = Map("ETag" -> etag))
+ }
+
+ withClue(s"HTTP $status: ") {
+ LakeFSStorageClient.put(
+ "x".getBytes(UTF_8),
+ 1,
+ s"$stubBaseUrl$partPath",
+ 2
+ ) shouldEqual etag
+ }
+ }
+ }
+
+ it should "name the part and the status when the object store rejects the upload" in {
+ // 403 is the live failure: an expired or mis-signed URL. 204 and 307 are the shape check — the
+ // guard has to stay "exactly 200 or 201" rather than widen to "4xx and worse", because neither
+ // of those two carries an ETag: accepting one books a part that was never stored, and the
+ // completion call then either NPEs on the missing ETag or assembles an object with a hole in it.
+ // 307 in particular is what an S3 region/endpoint mismatch answers.
+ Seq(403, 204, 307).foreach { status =>
+ requests.clear()
+ val partPath = s"/presigned/part-$status"
+ stub {
+ case StubRequest("PUT", p, _, _) if p == partPath => StubResponse(status)
+ }
+
+ val ex = intercept[RuntimeException] {
+ LakeFSStorageClient.put("x".getBytes(UTF_8), 1, s"$stubBaseUrl$partPath", 4)
+ }
+ // A multipart upload fails one part at a time; without the part number and the status in the
+ // message there is nothing in the log to say which part failed or whether it is retryable.
+ withClue(s"HTTP $status: ") {
+ ex.getMessage shouldEqual s"Part 4 upload failed (HTTP $status)"
+ }
+ }
+ }
+
// ---------------------------------------------------------------------------------------------
// Branch and object mutations
// ---------------------------------------------------------------------------------------------