You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "eugenegujing (via GitHub)" <gi...@apache.org> on 2026/06/12 20:04:06 UTC

[PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

eugenegujing opened a new pull request, #5643:
URL: https://github.com/apache/texera/pull/5643

   ### What changes were proposed in this PR?
   
   Adds a configurable scheduled cleanup job to file-service that reclaims the storage held by uploads that were started (or staged) but never committed.
   
   **Problem.** Files uploaded to a dataset sit in LakeFS staging until the user commits a version. LakeFS GC only manages committed data, and Texera has no background cleanup, so abandoned uploads accumulate forever in MinIO/S3, along with their `DATASET_UPLOAD_SESSION` rows. The only existing mitigations are the 24h presigned-URL expiry and the lazy re-init cleanup in `initMultipartUpload`, neither of which removes staged data for datasets nobody touches again.
   
   **Design.**
   
   ```
   FileService.run() ──(if enabled)──> Managed StagedFileCleanupJob
      every interval-minutes, on one daemon thread:
      ├─ path 1: DATASET_UPLOAD_SESSION rows with created_at < now − retention
      │          → abort LakeFS multipart, then delete row (parts cascade)
      ├─ path 2: per dataset repo, staged objects with mtime < now − retention
      │          → branch-reset via existing LakeFS API
      └─ audit log: sessions deleted / objects reset / errors
   ```
   
   Safety properties (each independently enforced):
   
   | Guard | Mechanism |
   |---|---|
   | Committed data is untouchable | the job only calls staging-scoped APIs (`diffBranch`, branch reset, multipart abort) — committed objects never appear in its inputs |
   | No race with an in-flight upload | staged objects whose path belongs to a non-expired session are excluded (exact complement of the deletion cutoff); default retention 72h is far above the 24h `PHYSICAL_ADDRESS_EXPIRATION_TIME_HRS`, so deleted sessions are already non-resumable |
   | Crash-safe ordering | multipart abort happens before the row delete; a crash in between leaves the row, which is retried next round (abort 404 = already aborted) |
   | Idempotent & isolated | every action is safely repeatable; per-item failures are logged, counted in the report, and never abort the batch; staged deletions and orphaned repos (dataset row without a live LakeFS repo) are skipped instead of erroring every round |
   | Operator control | `storage.cleanup.enabled` / `retention-hours` / `interval-minutes`, each overridable via `STORAGE_CLEANUP_*` env vars; the whole feature can be disabled |
   
   Also in this PR: file-service test suites now fork one JVM per suite (`build.sbt`). Each testcontainers-based suite boots its own LakeFS/MinIO/Postgres stack and mutates JVM-wide singletons (`StorageConfig` endpoints), so sharing one JVM broke whichever container suite ran second; forking isolates the Docker client and singleton state. `LakeFSStorageClient` gains one helper (`getStagedObjectMtime`) because the diff API carries no timestamps.
   
   Feedback welcome on the default retention (72h ≈ covers a weekend; it is a config default, not a hard-coded value).
   
   > **Note:** since this change automatically deletes user-uploaded data, it touches a fairly high-risk area. I have been deliberately conservative (staging-scoped APIs only, generous retention, kill switch, audit logging), but I'm happy to discuss any part of the design — including making the feature opt-in (`enabled = false` by default) if that feels safer for a first release.
   
   ### Any related issues, documentation, discussions?
   
   Closes #3681.
   
   ### How was this PR tested?
   
   New `StagedFileCleanupJobSpec` (11 tests, real LakeFS/MinIO/Postgres via testcontainers + embedded Postgres for the DAO): expired session deleted with part-row cascade; fresh session survives; expired staged object reset while a committed object survives; fresh staged object survives; idempotent second run reports zeros; active upload and its staged file untouched; mixed-batch exact counts; ±1min boundary around the cutoff; out-of-band-aborted multipart treated as already-aborted (404 rule); per-item failure counted without aborting the batch.
   
   ```
   sbt "FileService/test"
   ```
   
   111 tests, 4 suites, all passed (includes the pre-existing `DatasetResourceSpec` 93 tests — no regression). Also ran `sbt scalafixAll` and `sbt scalafmtAll` (clean). Requires Docker for the testcontainers suites.
   
   ### Was this PR authored or co-authored using generative AI tooling?
   
   Co-authored by: Claude Code (Claude Fable 5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4705066156

   @eugenegujing please also fill up the tests cases. https://github.com/apache/texera/pull/5643#issuecomment-4694904609 the current coverage is quite low with a few files 0% patch coverage


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4702174476

   @carloea2 @xuang7 @aicam : please chime in as you did a related project before.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang merged PR #5643:
URL: https://github.com/apache/texera/pull/5643


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4786098185

   @Yicong-Huang Please check it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4804979483

   Please test [.../scala/org/apache/texera/service/FileService.scala](https://app.codecov.io/gh/apache/texera/pull/5643?src=pr&el=tree&filepath=file-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2FFileService.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZmlsZS1zZXJ2aWNlL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvdGV4ZXJhL3NlcnZpY2UvRmlsZVNlcnZpY2Uuc2NhbGE=), that's a core logic of this PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4694904609

   ## [Codecov](https://app.codecov.io/gh/apache/texera/pull/5643?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 52.51%. Comparing base ([`1572edf`](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5248032`](https://app.codecov.io/gh/apache/texera/commit/52480328119a54b62540729fd089487cd3699ea0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 11 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##               main    #5643   +/-   ##
   =========================================
     Coverage     52.51%   52.51%           
     Complexity     2484     2484           
   =========================================
     Files          1071     1071           
     Lines         41363    41363           
     Branches       4441     4441           
   =========================================
     Hits          21720    21720           
     Misses        18371    18371           
     Partials       1272     1272           
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
   |---|---|---|---|
   | [access-control-service](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.61% <ø> (ø)` | | |
   | [agent-service](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.36% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [amber](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.30% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `1.65% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [config-service](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.71% <ø> (ø)` | | |
   | [file-service](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.21% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [frontend](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.10% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pyamber](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.72% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [python](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.75% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/5643/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.69% <ø> (ø)` | | Carriedforward from [1572edf](https://app.codecov.io/gh/apache/texera/commit/1572edf43f708a89573710a4aab9e06726a33924?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   *This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/5643?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4715653722

   Hi @eugenegujing i actually think it's fine to keep the PR as it is with minor fixes. The persisted audit history maybe an overkill: we can simply log it out. Also admin would be headache if needs to periodically confirm the deletion. 
   
   It's hard to balance over defensive code and reckless code. Just be careful about destructive operations in general, moving forward as well.
   
   Also, a nit: let's make the pr description a bit concise. Also some of the comments can be a bit shorter to make communication more efficient.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4701687366

   @Yicong-Huang Thank you for the detailed review. It was really helpful. I've tried to address all three points in the latest commit. Here's what I did, please let me know if I've misunderstood anything.
   
   **On combining the steps in a transaction (comments 1 & 3):** I've wrapped Path 1 so that, for each session, the row delete and the multipart abort run in one `SqlServer.withTransaction` block, with the delete staged first. My thinking is that if the abort then fails, the transaction rolls back and the row survives, so we shouldn't end up in the partial state you described (record gone but the LakeFS entry left behind), so it'd just be retried next round. Please correct me if there's a case I'm missing here.
   
   One thing I wasn't sure how to fully solve: since LakeFS is an external service, I don't think it can really take part in a Postgres transaction (a DB rollback can't undo an abort that already went through), so I couldn't get true all-or-nothing across both systems. What I did instead was lean on ordering + idempotency. The DB write is gated on the abort, and the abort seems to be idempotent (re-aborting an already-aborted upload returns 404, which we treat as success), so in my understanding a successful abort followed by a later failure should self-heal on the next round. I also kept the transaction per-session rather than per-round, mainly so we don't hold a DB connection open across many LakeFS calls, but I'm happy to change the granularity if you'd prefer. For Path 2 (resetting uncommitted objects) there's no DB write, so I didn't see anything to wrap there; it also runs as an independent sweep that doesn't rely on the session records, which I think gives us a second chance t
 o clean any orphaned object even if its row is already gone.
   
   **On testing the different step failures (comment 2):** I added tests for each case you mentioned, checking the result is either rolled back or cleaned next round: no DB record (orphan / null-`repository_name` session), no file found (already-aborted 404 and repo-level 404), and a timeout / generic error (non-404 abort failure → delete rolled back, row survives, retried). I also added one for "cleaned on the next round" (a transient failure self-healing) and one for failure isolation (a failing item not blocking a healthy one in the same round). The only one I couldn't reproduce deterministically is the per-object 404 in Path 2. It's handled the same way in code, but I couldn't trigger it in a container test without faking the client, so I left a comment pointing to the equivalent repo-level 404 test instead. Let me know if you'd like me to cover that differently.
   
   Thanks again for that and I'm happy to adjust any of this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3483083807


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {

Review Comment:
   Resolved. runCleanupOnce is private[util]



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3483071417


##########
common/config/src/main/resources/storage.conf:
##########
@@ -133,6 +133,22 @@ storage {
         }
     }
 
+    # Automated cleanup of uploaded but uncommitted dataset files
+    cleanup {
+        enabled = true

Review Comment:
   storage.cleanup.enabled now defaults to false



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4699812931

   @Yicong-Huang I have added 7 more tests for this. Could you please review it again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4813769083

   @Yicong-Huang Hi Yicong, I Added tests covering the cleanup-job registration (enabled/disabled + invalid-config guard + fail-fast) for FileServiceSpec, and tests covering the TOCTOU mtime re-read guard (skips reset when the re-read is fresh, resets when still expired), and the scheduled tick's exception-swallowing for StagedFileCleanupJobSpec.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4783406962

   <!-- texera-reviewer-suggestion -->
   ### Automated Reviewer Suggestions
   
   Based on the `git blame` history of the changed files, we recommend the following reviewers:
   
   - **Committers with relevant context:** `@pjfanning`
     You can request their reviews formally with `/request-review @pjfanning`.
   
   - **Contributors with relevant context:** `@kunwp1`, `@xuang7`, `@aglinxinyuan`
     You can notify them by mentioning `@kunwp1`, `@xuang7`, `@aglinxinyuan` in a comment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4804329202

   @Yicong-Huang Hi Yicong, I think I have already fixed those conflicts. Could you please check it again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3483557236


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList
+
+    expiredSessions.foreach { session =>
+      try {
+        // Delete the row and abort the multipart in one transaction, deleting FIRST. LakeFS is
+        // external and cannot truly enroll in a DB transaction, but the abort is idempotent
+        // (re-aborting an already-aborted upload returns 404, treated as success below), so the
+        // only risk is the abort failing AFTER the delete is staged. By staging the delete first
+        // and letting a non-404 abort failure roll the whole transaction back, the session row
+        // survives and the next round retries — never leaving an orphaned multipart behind.
+        SqlServer.withTransaction(ctx) { txn =>
+          txn
+            .deleteFrom(DATASET_UPLOAD_SESSION)
+            .where(DATASET_UPLOAD_SESSION.UPLOAD_ID.eq(session.getUploadId))
+            .execute()
+          repoNameByDid.get(session.getDid) match {
+            case Some(repoName) =>
+              try {
+                LakeFSStorageClient.abortPresignedMultipartUploads(
+                  repoName,
+                  session.getFilePath,
+                  session.getUploadId,
+                  session.getPhysicalAddress
+                )
+              } catch {
+                // Already aborted (or never materialized): safe to delete the session row.
+                case e: ApiException if e.getCode == 404 =>
+                  logger.debug(
+                    s"Multipart upload ${session.getUploadId} not found in LakeFS; " +
+                      "treating as already aborted"
+                  )
+              }
+            case None =>
+              // Dataset row gone or repository_name is NULL: the multipart lived in that
+              // repository's namespace, so there is nothing left to abort.
+              logger.debug(
+                s"No repository for dataset ${session.getDid}; " +
+                  s"deleting orphan upload session ${session.getUploadId}"
+              )
+          }
+        }
+        sessionsDeleted += 1
+      } catch {
+        case t: Throwable =>
+          logger.warn(
+            s"Failed to clean up upload session ${session.getUploadId} " +
+              s"(did=${session.getDid}, path=${session.getFilePath}); will retry next round",
+            t
+          )
+          errors += 1
+      }
+    }
+
+    // File paths of still-active (non-expired) upload sessions, per dataset. Staged objects
+    // belonging to an active upload must never be reset.
+    val activePathsByDid: Map[Integer, Set[String]] = ctx
+      .select(DATASET_UPLOAD_SESSION.DID, DATASET_UPLOAD_SESSION.FILE_PATH)
+      .from(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.ge(cutoff))
+      .fetch()
+      .asScala
+      .groupBy(record => record.get(DATASET_UPLOAD_SESSION.DID))
+      .map {
+        case (did, records) =>
+          did -> records.map(_.get(DATASET_UPLOAD_SESSION.FILE_PATH)).toSet
+      }
+
+    // Path 2: reset staged (uncommitted) objects older than the retention window.
+    repoNameByDid.foreach {
+      case (did, repoName) =>
+        try {
+          val activePaths = activePathsByDid.getOrElse(did, Set.empty)
+          val stagedObjects = LakeFSStorageClient.retrieveUncommittedObjects(repoName)
+          // diffBranch carries no mtime, so each candidate costs one extra statObject call
+          // (N+1). Unavoidable until LakeFS exposes timestamps in the diff API.
+          stagedObjects.foreach { diff =>
+            val path = diff.getPath

Review Comment:
   I dropped the upfront snapshot and now re-check both guards right before reset (live hasActiveUploadSession query + a re-read of mtime), shrinking the window from the whole path2 loop down to the gap between the final statObject and resetObject call



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3411268739


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList

Review Comment:
   how large this list can be, realistically? or in another word, how many uncommitted datasets are usually accumulated in 72 hours? just don't want this to OOM if fetch a big table. 



##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {

Review Comment:
   this method is not supposed to be invoked from outside right? can we make it `private[util]`?



##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList
+
+    expiredSessions.foreach { session =>
+      try {
+        // Delete the row and abort the multipart in one transaction, deleting FIRST. LakeFS is
+        // external and cannot truly enroll in a DB transaction, but the abort is idempotent
+        // (re-aborting an already-aborted upload returns 404, treated as success below), so the
+        // only risk is the abort failing AFTER the delete is staged. By staging the delete first
+        // and letting a non-404 abort failure roll the whole transaction back, the session row
+        // survives and the next round retries — never leaving an orphaned multipart behind.
+        SqlServer.withTransaction(ctx) { txn =>
+          txn
+            .deleteFrom(DATASET_UPLOAD_SESSION)
+            .where(DATASET_UPLOAD_SESSION.UPLOAD_ID.eq(session.getUploadId))
+            .execute()
+          repoNameByDid.get(session.getDid) match {
+            case Some(repoName) =>
+              try {
+                LakeFSStorageClient.abortPresignedMultipartUploads(
+                  repoName,
+                  session.getFilePath,
+                  session.getUploadId,
+                  session.getPhysicalAddress
+                )
+              } catch {
+                // Already aborted (or never materialized): safe to delete the session row.
+                case e: ApiException if e.getCode == 404 =>
+                  logger.debug(
+                    s"Multipart upload ${session.getUploadId} not found in LakeFS; " +
+                      "treating as already aborted"
+                  )
+              }
+            case None =>
+              // Dataset row gone or repository_name is NULL: the multipart lived in that
+              // repository's namespace, so there is nothing left to abort.
+              logger.debug(
+                s"No repository for dataset ${session.getDid}; " +
+                  s"deleting orphan upload session ${session.getUploadId}"
+              )
+          }
+        }
+        sessionsDeleted += 1
+      } catch {
+        case t: Throwable =>
+          logger.warn(
+            s"Failed to clean up upload session ${session.getUploadId} " +
+              s"(did=${session.getDid}, path=${session.getFilePath}); will retry next round",
+            t
+          )
+          errors += 1
+      }
+    }
+
+    // File paths of still-active (non-expired) upload sessions, per dataset. Staged objects
+    // belonging to an active upload must never be reset.
+    val activePathsByDid: Map[Integer, Set[String]] = ctx
+      .select(DATASET_UPLOAD_SESSION.DID, DATASET_UPLOAD_SESSION.FILE_PATH)
+      .from(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.ge(cutoff))
+      .fetch()
+      .asScala
+      .groupBy(record => record.get(DATASET_UPLOAD_SESSION.DID))
+      .map {
+        case (did, records) =>
+          did -> records.map(_.get(DATASET_UPLOAD_SESSION.FILE_PATH)).toSet
+      }
+
+    // Path 2: reset staged (uncommitted) objects older than the retention window.
+    repoNameByDid.foreach {
+      case (did, repoName) =>
+        try {
+          val activePaths = activePathsByDid.getOrElse(did, Set.empty)
+          val stagedObjects = LakeFSStorageClient.retrieveUncommittedObjects(repoName)
+          // diffBranch carries no mtime, so each candidate costs one extra statObject call
+          // (N+1). Unavoidable until LakeFS exposes timestamps in the diff API.
+          stagedObjects.foreach { diff =>
+            val path = diff.getPath

Review Comment:
   can this happen:
   
   ```
     T0  snapshot activePathsByDid                    // P not in active set (its old session was just cleaned in path 1)
     T1  getStagedObjectMtime(P) -> old mtime
     T2  user starts a new session for P and finishes its multipart   // V2 lands on the branch
     T3  job sees mtime < cutoff (still using the stale read from T1)
         -> resetObjectUploadOrDeletion(P) wipes V2
   ```
   
   Trigger conditions:
     - The old session for P was cleaned in path 1 earlier in the same round, so P is absent from activePathsByDid.
     - A new upload session for P is created after the activePathsByDid snapshot and completes its multipart before this iteration's reset call.
   
   The window is narrow given a 72h retention and a single-threaded job, but it's a real user-data-loss path: the user just finished an upload and the cleanup wipes it before commit. 
   
   I think it is due to we read the activePathsByDid and did not update it.  but technically this window is super small. 



##########
common/config/src/main/resources/storage.conf:
##########
@@ -133,6 +133,22 @@ storage {
         }
     }
 
+    # Automated cleanup of uploaded but uncommitted dataset files
+    cleanup {
+        enabled = true

Review Comment:
   do you think make it default to false is safer? if default to true, after you merge to main all developer's env will start to delete uncommitted files. and if there  is a bug we could delete real user's datasets mistakenly in prod. that could be destructive. 
   
   maybe better to default to false, then after merge, turn it on explicitly on different prod deployment. later if we confirm the logic is correct later, we can make it default true. 



##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList
+
+    expiredSessions.foreach { session =>
+      try {
+        // Delete the row and abort the multipart in one transaction, deleting FIRST. LakeFS is
+        // external and cannot truly enroll in a DB transaction, but the abort is idempotent
+        // (re-aborting an already-aborted upload returns 404, treated as success below), so the
+        // only risk is the abort failing AFTER the delete is staged. By staging the delete first
+        // and letting a non-404 abort failure roll the whole transaction back, the session row
+        // survives and the next round retries — never leaving an orphaned multipart behind.
+        SqlServer.withTransaction(ctx) { txn =>
+          txn
+            .deleteFrom(DATASET_UPLOAD_SESSION)
+            .where(DATASET_UPLOAD_SESSION.UPLOAD_ID.eq(session.getUploadId))
+            .execute()
+          repoNameByDid.get(session.getDid) match {
+            case Some(repoName) =>
+              try {
+                LakeFSStorageClient.abortPresignedMultipartUploads(
+                  repoName,
+                  session.getFilePath,
+                  session.getUploadId,
+                  session.getPhysicalAddress
+                )
+              } catch {
+                // Already aborted (or never materialized): safe to delete the session row.
+                case e: ApiException if e.getCode == 404 =>
+                  logger.debug(
+                    s"Multipart upload ${session.getUploadId} not found in LakeFS; " +
+                      "treating as already aborted"
+                  )
+              }
+            case None =>
+              // Dataset row gone or repository_name is NULL: the multipart lived in that
+              // repository's namespace, so there is nothing left to abort.
+              logger.debug(
+                s"No repository for dataset ${session.getDid}; " +
+                  s"deleting orphan upload session ${session.getUploadId}"
+              )
+          }
+        }
+        sessionsDeleted += 1
+      } catch {
+        case t: Throwable =>
+          logger.warn(
+            s"Failed to clean up upload session ${session.getUploadId} " +
+              s"(did=${session.getDid}, path=${session.getFilePath}); will retry next round",
+            t
+          )
+          errors += 1
+      }
+    }
+
+    // File paths of still-active (non-expired) upload sessions, per dataset. Staged objects
+    // belonging to an active upload must never be reset.
+    val activePathsByDid: Map[Integer, Set[String]] = ctx
+      .select(DATASET_UPLOAD_SESSION.DID, DATASET_UPLOAD_SESSION.FILE_PATH)
+      .from(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.ge(cutoff))
+      .fetch()
+      .asScala
+      .groupBy(record => record.get(DATASET_UPLOAD_SESSION.DID))
+      .map {
+        case (did, records) =>
+          did -> records.map(_.get(DATASET_UPLOAD_SESSION.FILE_PATH)).toSet
+      }
+
+    // Path 2: reset staged (uncommitted) objects older than the retention window.
+    repoNameByDid.foreach {
+      case (did, repoName) =>
+        try {
+          val activePaths = activePathsByDid.getOrElse(did, Set.empty)
+          val stagedObjects = LakeFSStorageClient.retrieveUncommittedObjects(repoName)
+          // diffBranch carries no mtime, so each candidate costs one extra statObject call
+          // (N+1). Unavoidable until LakeFS exposes timestamps in the diff API.
+          stagedObjects.foreach { diff =>
+            val path = diff.getPath
+            val isObjectWrite =
+              diff.getType == Diff.TypeEnum.ADDED || diff.getType == Diff.TypeEnum.CHANGED
+            if (!isObjectWrite) {
+              // E.g. a staged deletion of a committed file: there is no object behind it and
+              // it consumes no storage, so leaving it is correct and cheap.
+              logger.debug(s"Skipping staged ${diff.getType} entry '$path' in '$repoName'")
+            } else if (!activePaths.contains(path)) {
+              try {
+                val mtime = LakeFSStorageClient.getStagedObjectMtime(repoName, path)
+                if (mtime < cutoff.toEpochSecond) {
+                  LakeFSStorageClient.resetObjectUploadOrDeletion(repoName, path)
+                  objectsReset += 1
+                }
+              } catch {
+                // Concurrently committed/reset, or already cleaned by another round: the
+                // object is gone, which is the desired end state for an idempotent job.
+                case e: ApiException if e.getCode == 404 =>
+                  logger.debug(
+                    s"Staged object '$path' not found in repo '$repoName'; " +
+                      "treating as already cleaned"
+                  )
+                case t: Throwable =>
+                  logger.warn(
+                    s"Failed to clean up staged object '$path' in repo '$repoName'",
+                    t
+                  )
+                  errors += 1
+              }
+            }
+          }
+        } catch {
+          // The dataset's LakeFS repository was deleted out-of-band (a supported state):
+          // nothing staged to clean up there.
+          case e: ApiException if e.getCode == 404 =>
+            logger.debug(s"Repository '$repoName' not found in LakeFS; skipping")
+          case t: Throwable =>
+            logger.warn(s"Failed to clean up staged objects in repo '$repoName'", t)
+            errors += 1
+        }
+    }
+
+    logger.info(

Review Comment:
   nit: change this to debug level?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4704880761

   I will check. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4785296972

   @eugenegujing thanks. please resolve conflicts. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4804981620

   btw, it will be good if you can reply to our comments inline so we know what have you changed. it is a bit hard to track the changes you made.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4696997650

   @eugenegujing thanks for the PR. Please add more tests to guard the behavior.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3408623535


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  private var executor: ScheduledExecutorService = _

Review Comment:
   Add parameter validation for retentionHours/intervalMinutes. As-is, a misconfiguration (<= 0) can cause immediate mass deletion (retentionHours <= 0) or throw during scheduling (intervalMinutes <= 0), turning a config typo into either data loss or a startup failure.



##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList
+
+    expiredSessions.foreach { session =>
+      try {
+        repoNameByDid.get(session.getDid) match {
+          case Some(repoName) =>
+            try {
+              LakeFSStorageClient.abortPresignedMultipartUploads(
+                repoName,
+                session.getFilePath,
+                session.getUploadId,
+                session.getPhysicalAddress
+              )
+            } catch {
+              // Already aborted (or never materialized): safe to delete the session row.
+              case e: ApiException if e.getCode == 404 =>
+                logger.debug(
+                  s"Multipart upload ${session.getUploadId} not found in LakeFS; " +
+                    "treating as already aborted"
+                )
+            }
+          case None =>
+            // Dataset row gone or repository_name is NULL: the multipart lived in that
+            // repository's namespace, so there is nothing left to abort.
+            logger.debug(
+              s"No repository for dataset ${session.getDid}; " +
+                s"deleting orphan upload session ${session.getUploadId}"
+            )
+        }
+        ctx
+          .deleteFrom(DATASET_UPLOAD_SESSION)
+          .where(DATASET_UPLOAD_SESSION.UPLOAD_ID.eq(session.getUploadId))
+          .execute()
+        sessionsDeleted += 1
+      } catch {
+        case t: Throwable =>
+          logger.warn(
+            s"Failed to clean up upload session ${session.getUploadId} " +
+              s"(did=${session.getDid}, path=${session.getFilePath}); will retry next round",
+            t
+          )
+          errors += 1
+      }
+    }
+
+    // File paths of still-active (non-expired) upload sessions, per dataset. Staged objects
+    // belonging to an active upload must never be reset.
+    val activePathsByDid: Map[Integer, Set[String]] = ctx
+      .select(DATASET_UPLOAD_SESSION.DID, DATASET_UPLOAD_SESSION.FILE_PATH)
+      .from(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.ge(cutoff))
+      .fetch()
+      .asScala
+      .groupBy(record => record.get(DATASET_UPLOAD_SESSION.DID))
+      .map {
+        case (did, records) =>
+          did -> records.map(_.get(DATASET_UPLOAD_SESSION.FILE_PATH)).toSet
+      }
+
+    // Path 2: reset staged (uncommitted) objects older than the retention window.
+    repoNameByDid.foreach {
+      case (did, repoName) =>
+        try {
+          val activePaths = activePathsByDid.getOrElse(did, Set.empty)
+          val stagedObjects = LakeFSStorageClient.retrieveUncommittedObjects(repoName)
+          // diffBranch carries no mtime, so each candidate costs one extra statObject call
+          // (N+1). Unavoidable until LakeFS exposes timestamps in the diff API.
+          stagedObjects.foreach { diff =>
+            val path = diff.getPath
+            val isObjectWrite =
+              diff.getType == Diff.TypeEnum.ADDED || diff.getType == Diff.TypeEnum.CHANGED
+            if (!isObjectWrite) {
+              // E.g. a staged deletion of a committed file: there is no object behind it and
+              // it consumes no storage, so leaving it is correct and cheap.
+              logger.debug(s"Skipping staged ${diff.getType} entry '$path' in '$repoName'")
+            } else if (!activePaths.contains(path)) {
+              try {
+                val mtime = LakeFSStorageClient.getStagedObjectMtime(repoName, path)
+                if (mtime < cutoff.toEpochSecond) {
+                  LakeFSStorageClient.resetObjectUploadOrDeletion(repoName, path)
+                  objectsReset += 1
+                }
+              } catch {
+                case t: Throwable =>
+                  logger.warn(
+                    s"Failed to clean up staged object '$path' in repo '$repoName'",
+                    t
+                  )
+                  errors += 1
+              }

Review Comment:
   Treat ApiException 404 from stat/reset of a staged object as a successful no-op. Without this, concurrent activity (another cleanup round, user commit/reset) can turn an already-cleaned path into noisy warnings and increment `errors`, even though the job is intended to be idempotent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4703189650

   @Yicong-Huang did @eugenegujing addressed all your comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4705572243

   @Yicong-Huang Thanks! I think your idea helps a lot. I'm thinking of transforming this PR from deleting any uncommitted trash files to delivering an audit summary over a retention window instead.
   
   For this PR, I am considering the safest first step:
   
   A. Audit-only scheduled scan
   - The job still scans expired upload sessions and stale uncommitted LakeFS objects using the same candidate-detection logic that cleanup would need.
   - It does not abort multipart uploads, delete DB rows, or reset LakeFS objects.
   - It only emits a round summary, for example: expiredSessionsFound, staleObjectsFound, errors, truncated.
   - Candidate-level details can stay at debug level so we can inspect them when needed without making normal logs noisy.
   - Cleanup stays disabled / non-destructive by default.
   
   The important connection is that A is not a throwaway version: it establishes the candidate model and the scheduled scan path, but makes the output observational only. Then follow-up PRs can gradually evolve the same audit output into B:
   
   B. Persisted audit history + optional cleanup action
   - Persist audit runs and candidates in DB so admins can inspect historical trends and exact candidates.
   - Build on the same fields emitted by the audit summary in A, instead of introducing a separate cleanup path later.
   - Add an explicit admin-facing cleanup action or config-gated cleanup mode only after the persisted audit flow exists.
   - If real deletion is added, it should be opt-in and reviewed in a separate PR, not enabled by default.
   
   So this PR would focus on non-destructive observability first, while keeping the implementation shaped so later PRs can step-by-step develop it into persisted audit history and, eventually, an explicit cleanup action. What do you think?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4694917522

   <!-- texera-benchmarks-comment -->
   ## ✅ No material benchmark regressions detected
   
   🟢 2 better · 🔴 0 worse · ⚪ 13 noise (<±5%) · 0 without baseline
   
   > CI benchmark results are noisy; treat <±5% as noise unless repeated.
   
   [Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/27439945506)
   
   | | config | throughput | MB/s | latency | max Δ latest / 7d |
   |---|---|---:|---:|---:|---:|
   | ⚪ | bs=10 sw=10 sl=64 | 442 | 0.27 | 21,950/33,520/33,520 us | ⚪ within ±5% / 🟢 +10.9% |
   | 🟢 | bs=100 sw=10 sl=64 | 927 | 0.566 | 106,848/130,618/130,618 us | 🟢 -18.5% / 🟢 -5.1% |
   | ⚪ | bs=1000 sw=10 sl=64 | 1,116 | 0.681 | 896,608/948,212/948,212 us | ⚪ within ±5% / 🟢 +8.5% |
   
   <details><summary>Baseline details</summary>
   
   Latest main `227cbd7` from 2026-06-12T19:07:43.071Z
   
   | config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
   |---|---|---:|---:|---:|---:|---:|
   | bs=10 sw=10 sl=64 | throughput | 442 tuples/sec | 422.14 tuples/sec | 398.78 tuples/sec | +4.7% | +10.8% |
   | bs=10 sw=10 sl=64 | MB/s | 0.27 MB/s | 0.258 MB/s | 0.243 MB/s | +4.8% | +10.9% |
   | bs=10 sw=10 sl=64 | p50 | 21,950 us | 22,135 us | 24,396 us | -0.8% | -10.0% |
   | bs=10 sw=10 sl=64 | p95 | 33,520 us | 34,198 us | 35,892 us | -2.0% | -6.6% |
   | bs=10 sw=10 sl=64 | p99 | 33,520 us | 34,198 us | 35,892 us | -2.0% | -6.6% |
   | bs=100 sw=10 sl=64 | throughput | 927 tuples/sec | 894.49 tuples/sec | 886.25 tuples/sec | +3.6% | +4.6% |
   | bs=100 sw=10 sl=64 | MB/s | 0.566 MB/s | 0.546 MB/s | 0.541 MB/s | +3.7% | +4.6% |
   | bs=100 sw=10 sl=64 | p50 | 106,848 us | 107,334 us | 112,613 us | -0.5% | -5.1% |
   | bs=100 sw=10 sl=64 | p95 | 130,618 us | 160,327 us | 135,197 us | -18.5% | -3.4% |
   | bs=100 sw=10 sl=64 | p99 | 130,618 us | 160,327 us | 135,197 us | -18.5% | -3.4% |
   | bs=1000 sw=10 sl=64 | throughput | 1,116 tuples/sec | 1,077 tuples/sec | 1,029 tuples/sec | +3.6% | +8.5% |
   | bs=1000 sw=10 sl=64 | MB/s | 0.681 MB/s | 0.657 MB/s | 0.628 MB/s | +3.6% | +8.4% |
   | bs=1000 sw=10 sl=64 | p50 | 896,608 us | 928,059 us | 976,698 us | -3.4% | -8.2% |
   | bs=1000 sw=10 sl=64 | p95 | 948,212 us | 950,455 us | 1,032,282 us | -0.2% | -8.1% |
   | bs=1000 sw=10 sl=64 | p99 | 948,212 us | 950,455 us | 1,032,282 us | -0.2% | -8.1% |
   
   </details>
   
   <details><summary>Raw CSV</summary>
   
   ```csv
   config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
   0,10,10,64,20,452.53,200,128000,442,0.270,21950.40,33520.21,33520.21
   1,100,10,64,20,2157.40,2000,1280000,927,0.566,106848.09,130618.48,130618.48
   2,1000,10,64,20,17928.33,20000,12800000,1116,0.681,896607.85,948211.50,948211.50
   ```
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3408645419


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })

Review Comment:
   if we do run this on a seperate thread (which I think it is the right way to go), then we have to be careful for the destructive operations fired in this thread will not affect other threads (e.g., creating, uploading dataset). 
   
   We need to make sure two things
   - all the DB operations in this thread can be atomic, and won't create a partial state (e.g., by using some DB transactions), but maybe hard to deal with lakefs, AND
   - if any operation fails and left a partial state, the partial state won't affect other threads, and also the partial state can be continued to cleaned in the next round. 
   
   More concrete example: if we removed the  session record, then deleting on lakefs failed due to some weird reason, it could result in:
   1. **Affect other threads:** The next time user wants  to create a new  dataset with the same name, there is no sessiion record so we allowed user to create, but it always failed because lakefs entry is still there.
   2. **Uncleaned partial state:** The lakefs entry will always stay and never be removed, because we don't know there is an untracked lakefs entry as the sessions record is gone.



##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap

Review Comment:
   ok, as you are operating multiple actions on DB, including deletion, please combine these steps in one transaction:
   
   - repoName look up
   - abort partial upload (I assume this also writes to lakefs's catalog)
   - delete session record
   - active path look up
   - reset uncommitted objects
   
   So that any steps failed can abort and roll back, otherwise it may get into a partial state. 



##########
file-service/src/test/scala/org/apache/texera/service/util/StagedFileCleanupJobSpec.scala:
##########
@@ -0,0 +1,716 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import io.lakefs.clients.sdk.ApiException
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.auth.SessionUser
+import org.apache.texera.dao.MockTexeraDB
+import org.apache.texera.dao.jooq.generated.enums.UserRoleEnum
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSessionPart.DATASET_UPLOAD_SESSION_PART
+import org.apache.texera.dao.jooq.generated.tables.daos.{DatasetDao, UserDao}
+import org.apache.texera.dao.jooq.generated.tables.pojos.{Dataset, User}
+import org.apache.texera.service.MockLakeFS
+import org.apache.texera.service.resource.DatasetResource
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import java.io.ByteArrayInputStream
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.time.OffsetDateTime
+import java.util.Optional
+
+/**
+  * Spec for [[StagedFileCleanupJob]] (issue #3681 — automated cleanup of uploaded but
+  * uncommitted files).
+  *
+  * Contract under test:
+  *   - `runCleanupOnce(now)` deletes DATASET_UPLOAD_SESSION rows whose created_at is older
+  *     than `retentionHours` relative to the injected `now` (aborting their LakeFS multipart
+  *     first; part rows go away via ON DELETE CASCADE),
+  *   - resets LakeFS staged (uncommitted) objects whose mtime exceeds retention,
+  *   - skips staged objects that belong to a non-expired upload session,
+  *   - never touches committed objects,
+  *   - counts per-item failures in `errors` without aborting the batch,
+  *   - is idempotent.
+  *
+  * Tests never sleep to age things: sessions are aged either by passing a future `now`
+  * (everything created "now" is then older than retention) or by writing an explicit
+  * created_at via jOOQ. Staged-object mtimes cannot be faked, so object-expiry tests pass a
+  * future `now`, and "fresh staged object" semantics under a future `now` are exercised via
+  * the session-protection rule (created_at moved next to the future `now`).
+  */
+class StagedFileCleanupJobSpec
+    extends AnyFlatSpec
+    with Matchers
+    with MockTexeraDB
+    with MockLakeFS
+    with BeforeAndAfterAll
+    with BeforeAndAfterEach {
+
+  // ---------------------------------------------------------------------------
+  // Job configuration under test
+  // ---------------------------------------------------------------------------
+  private val RetentionHours = 24
+  private val IntervalMinutes = 60
+
+  private lazy val job = new StagedFileCleanupJob(RetentionHours, IntervalMinutes)
+
+  /** A `now` far enough in the future that anything created at real wall-clock time is expired. */
+  private def farFuture: OffsetDateTime =
+    OffsetDateTime.now().plusHours(RetentionHours.toLong + 1L)
+
+  // ---------------------------------------------------------------------------
+  // Fixtures (minimal copies of the DatasetResourceSpec idioms)
+  // ---------------------------------------------------------------------------
+  private val ownerUser: User = {
+    val user = new User
+    user.setName("cleanup_test_user")
+    user.setPassword("123")
+    user.setEmail("cleanup_test_user@test.com")
+    user.setRole(UserRoleEnum.ADMIN)
+    user
+  }
+
+  private val repoName: String = s"cleanup-ds-${System.nanoTime()}"
+
+  private val cleanupDataset: Dataset = {
+    val dataset = new Dataset
+    dataset.setName("cleanup-ds")
+    dataset.setRepositoryName(repoName)
+    dataset.setIsPublic(true)
+    dataset.setIsDownloadable(true)
+    dataset.setDescription("dataset for staged-file cleanup tests")
+    dataset
+  }
+
+  /** Object committed once up-front; must survive every cleanup run (safety pin). */
+  private val PinnedCommittedPath = "pinned/committed-pin.bin"
+
+  private lazy val sessionUser = new SessionUser(ownerUser)
+  private lazy val datasetResource = new DatasetResource()
+
+  private var lakeFsReady = false
+
+  // ---------------------------------------------------------------------------
+  // Lifecycle
+  // ---------------------------------------------------------------------------
+  override protected def beforeAll(): Unit = {
+    super.beforeAll()
+
+    initializeDBAndReplaceDSLContext()
+
+    new UserDao(getDSLContext.configuration()).insert(ownerUser)
+    cleanupDataset.setOwnerUid(ownerUser.getUid)
+    new DatasetDao(getDSLContext.configuration()).insert(cleanupDataset)
+  }
+
+  // Containers (MockLakeFS) only become reachable after the suite starts running, so all
+  // LakeFS setup happens lazily here rather than in beforeAll — same reason
+  // DatasetResourceSpec initializes its repo in beforeEach.
+  override protected def beforeEach(): Unit = {
+    super.beforeEach()
+
+    if (!lakeFsReady) {
+      try LakeFSStorageClient.initRepo(repoName)
+      catch {
+        case e: ApiException if e.getCode == 409 => // already exists, fine
+      }
+      // Commit one object up-front: cleanup must NEVER touch committed objects.
+      LakeFSStorageClient.writeFileToRepo(
+        repoName,
+        PinnedCommittedPath,
+        new ByteArrayInputStream("pinned".getBytes(StandardCharsets.UTF_8))
+      )
+      LakeFSStorageClient.createCommit(repoName, "main", "pin committed object")
+      lakeFsReady = true
+    }
+
+    // Clean slate so report counts are exact and independent of test order.
+    // (Deliberately NOT done via the job under test, to keep fixtures independent of it.)
+    getDSLContext.deleteFrom(DATASET_UPLOAD_SESSION).execute()
+    LakeFSStorageClient
+      .retrieveUncommittedObjects(repoName)
+      .foreach(diff => LakeFSStorageClient.resetObjectUploadOrDeletion(repoName, diff.getPath))
+  }
+
+  override protected def afterAll(): Unit = {
+    try shutdownDB()
+    finally super.afterAll()
+  }
+
+  // ---------------------------------------------------------------------------
+  // Helpers
+  // ---------------------------------------------------------------------------
+  private def urlEnc(raw: String): String =
+    URLEncoder.encode(raw, StandardCharsets.UTF_8.name())
+
+  private def uniquePath(prefix: String): String =
+    s"$prefix/${System.nanoTime()}.bin"
+
+  /** Creates a real upload session (valid uploadId + physicalAddress) and returns its uploadId. */
+  private def initSession(filePath: String): String = {
+    val resp = datasetResource.multipartUpload(
+      "init",
+      ownerUser.getEmail,
+      cleanupDataset.getName,
+      urlEnc(filePath),
+      Optional.of(java.lang.Long.valueOf(16L)),
+      Optional.of(java.lang.Long.valueOf(32L)), // single part
+      Optional.empty(),
+      sessionUser
+    )
+    resp.getStatus shouldEqual 200
+    val record = fetchSession(filePath)
+    record should not be null
+    record.getUploadId
+  }
+
+  private def fetchSession(filePath: String) =
+    getDSLContext
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(
+        DATASET_UPLOAD_SESSION.UID
+          .eq(ownerUser.getUid)
+          .and(DATASET_UPLOAD_SESSION.DID.eq(cleanupDataset.getDid))
+          .and(DATASET_UPLOAD_SESSION.FILE_PATH.eq(filePath))
+      )
+      .fetchOne()
+
+  private def countPartRows(uploadId: String): Int =
+    getDSLContext
+      .selectCount()
+      .from(DATASET_UPLOAD_SESSION_PART)
+      .where(DATASET_UPLOAD_SESSION_PART.UPLOAD_ID.eq(uploadId))
+      .fetchOne(0, classOf[Int])
+
+  /** Pins a session's age precisely — the injectable-clock counterpart on the DB side. */
+  private def setSessionCreatedAt(uploadId: String, createdAt: OffsetDateTime): Unit =
+    getDSLContext
+      .update(DATASET_UPLOAD_SESSION)
+      .set(DATASET_UPLOAD_SESSION.CREATED_AT, createdAt)
+      .where(DATASET_UPLOAD_SESSION.UPLOAD_ID.eq(uploadId))
+      .execute()
+
+  /** Uploads an object to the repo branch WITHOUT committing (a staged/uncommitted object). */
+  private def stageObject(filePath: String, content: String = "staged-bytes"): Unit =
+    LakeFSStorageClient.writeFileToRepo(
+      repoName,
+      filePath,
+      new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8))
+    )
+
+  private def uncommittedPaths(): List[String] =
+    LakeFSStorageClient.retrieveUncommittedObjects(repoName).map(_.getPath)
+
+  private def committedPaths(): List[String] =
+    LakeFSStorageClient.retrieveObjectsOfVersion(repoName, "main").map(_.getPath)
+
+  /** Inserts a session row whose LakeFS multipart does not exist (forces a per-item failure). */
+  private def insertBogusSession(filePath: String): Unit =
+    getDSLContext
+      .insertInto(DATASET_UPLOAD_SESSION)
+      .set(DATASET_UPLOAD_SESSION.DID, cleanupDataset.getDid)
+      .set(DATASET_UPLOAD_SESSION.UID, ownerUser.getUid)
+      .set(DATASET_UPLOAD_SESSION.FILE_PATH, filePath)
+      .set(DATASET_UPLOAD_SESSION.UPLOAD_ID, s"bogus-upload-${System.nanoTime()}")
+      .set(
+        DATASET_UPLOAD_SESSION.PHYSICAL_ADDRESS,
+        "s3://nonexistent-bucket/nonexistent-key"
+      )
+      .set(DATASET_UPLOAD_SESSION.NUM_PARTS_REQUESTED, Int.box(1))
+      .set(DATASET_UPLOAD_SESSION.FILE_SIZE_BYTES, java.lang.Long.valueOf(16L))
+      .set(DATASET_UPLOAD_SESSION.PART_SIZE_BYTES, java.lang.Long.valueOf(32L))
+      .execute()
+
+  // ===========================================================================
+  // 1. Expired session is cleaned
+  // ===========================================================================
+  "StagedFileCleanupJob.runCleanupOnce" should "delete an expired upload session and its part rows" in {
+    val filePath = uniquePath("expired-session")
+    val uploadId = initSession(filePath)
+    countPartRows(uploadId) shouldEqual 1 // placeholder created at init
+
+    val report = job.runCleanupOnce(farFuture)
+
+    report.sessionsDeleted shouldEqual 1
+    report.errors shouldEqual 0
+    fetchSession(filePath) shouldBe null
+    countPartRows(uploadId) shouldEqual 0 // gone via ON DELETE CASCADE
+  }
+
+  // ===========================================================================
+  // 2. Fresh session survives
+  // ===========================================================================
+  it should "keep a fresh (non-expired) upload session" in {
+    val filePath = uniquePath("fresh-session")
+    val uploadId = initSession(filePath)
+
+    val report = job.runCleanupOnce(OffsetDateTime.now())
+
+    report.sessionsDeleted shouldEqual 0
+    report.errors shouldEqual 0
+    fetchSession(filePath) should not be null
+    countPartRows(uploadId) shouldEqual 1
+  }
+
+  // ===========================================================================
+  // 3. Expired staged object is reset (committed object untouched)
+  // ===========================================================================
+  it should "reset an expired staged object but never a committed one" in {
+    val stagedPath = uniquePath("expired-staged")
+    stageObject(stagedPath)
+    uncommittedPaths() should contain(stagedPath)
+
+    val report = job.runCleanupOnce(farFuture)
+
+    report.objectsReset should be >= 1
+    report.errors shouldEqual 0
+    uncommittedPaths() should not contain stagedPath
+    // safety pin: the committed object survives every cleanup
+    committedPaths() should contain(PinnedCommittedPath)
+  }
+
+  // ===========================================================================
+  // 4. Fresh staged object survives
+  // ===========================================================================
+  it should "keep a freshly staged object when run with the real current time" in {
+    val stagedPath = uniquePath("fresh-staged")
+    stageObject(stagedPath)
+
+    val report = job.runCleanupOnce(OffsetDateTime.now())
+
+    report.objectsReset shouldEqual 0
+    report.errors shouldEqual 0
+    uncommittedPaths() should contain(stagedPath)
+  }
+
+  // ===========================================================================
+  // 5. Idempotence
+  // ===========================================================================
+  it should "be idempotent: a second run with the same now reports all zeros" in {
+    val sessionPath = uniquePath("idempotent-session")
+    val stagedPath = uniquePath("idempotent-staged")
+    initSession(sessionPath)
+    stageObject(stagedPath)
+
+    val now = farFuture
+
+    val first = job.runCleanupOnce(now)
+    first.sessionsDeleted shouldEqual 1
+    first.objectsReset should be >= 1
+    first.errors shouldEqual 0
+
+    val second = job.runCleanupOnce(now)
+    second.sessionsDeleted shouldEqual 0
+    second.objectsReset shouldEqual 0
+    second.errors shouldEqual 0
+  }
+
+  // ===========================================================================
+  // 6. Active upload is not touched while other items expire
+  // ===========================================================================
+  it should "not touch a non-expired session or its staged object while expiring other items" in {
+    val now = farFuture
+
+    // Protected: a session that is fresh RELATIVE TO the injected now, with its staged
+    // file present on the branch. The skip rule must protect the object even though its
+    // real mtime is "older" than retention relative to the future now.
+    val protectedPath = uniquePath("active-upload")
+    val protectedUploadId = initSession(protectedPath)
+    stageObject(protectedPath)
+    setSessionCreatedAt(protectedUploadId, now.minusMinutes(5))
+
+    // Expirees: another session and an orphan staged object, both created at real now,
+    // i.e. older than retention relative to the future now.
+    val expiredSessionPath = uniquePath("expired-other-session")
+    initSession(expiredSessionPath)
+    val expiredStagedPath = uniquePath("expired-other-staged")
+    stageObject(expiredStagedPath)
+
+    val report = job.runCleanupOnce(now)
+
+    report.sessionsDeleted shouldEqual 1
+    report.objectsReset shouldEqual 1
+    report.errors shouldEqual 0
+
+    // survivors
+    fetchSession(protectedPath) should not be null
+    uncommittedPaths() should contain(protectedPath)
+    // expirees
+    fetchSession(expiredSessionPath) shouldBe null
+    uncommittedPaths() should not contain expiredStagedPath
+  }
+
+  // ===========================================================================
+  // 7. Report counting on a mixed batch
+  // ===========================================================================
+  it should "report exact counts for a mix of expired and fresh items" in {
+    val now = farFuture
+
+    // 2 expired sessions
+    val expired1 = uniquePath("mix-expired-1")
+    val expired2 = uniquePath("mix-expired-2")
+    initSession(expired1)
+    initSession(expired2)
+
+    // 1 fresh session protecting 1 fresh staged object (fresh relative to the injected now)
+    val freshPath = uniquePath("mix-fresh")
+    val freshUploadId = initSession(freshPath)
+    stageObject(freshPath)
+    setSessionCreatedAt(freshUploadId, now.minusMinutes(5))
+
+    // 1 expired staged object with no session
+    val expiredStaged = uniquePath("mix-expired-staged")
+    stageObject(expiredStaged)
+
+    val report = job.runCleanupOnce(now)
+
+    report.sessionsDeleted shouldEqual 2
+    report.objectsReset shouldEqual 1
+    report.errors shouldEqual 0
+
+    fetchSession(expired1) shouldBe null
+    fetchSession(expired2) shouldBe null
+    fetchSession(freshPath) should not be null
+    uncommittedPaths() should contain(freshPath)
+    uncommittedPaths() should not contain expiredStaged
+  }
+
+  // ===========================================================================
+  // 8. Retention boundary (precision via injectable clock + explicit created_at)
+  // ===========================================================================
+  it should "clean a session just past retention but keep one just inside it" in {
+    val now = OffsetDateTime.now()
+    val cutoff = now.minusHours(RetentionHours.toLong)
+
+    val survivorPath = uniquePath("boundary-survivor")
+    val survivorUploadId = initSession(survivorPath)
+    setSessionCreatedAt(survivorUploadId, cutoff.plusMinutes(1)) // retention - epsilon
+
+    val expiredPath = uniquePath("boundary-expired")
+    val expiredUploadId = initSession(expiredPath)
+    setSessionCreatedAt(expiredUploadId, cutoff.minusMinutes(1)) // retention + epsilon
+
+    val report = job.runCleanupOnce(now)
+
+    report.sessionsDeleted shouldEqual 1
+    report.errors shouldEqual 0
+    fetchSession(survivorPath) should not be null
+    fetchSession(expiredPath) shouldBe null
+  }
+
+  // ===========================================================================
+  // 9. Committed objects are never touched (dedicated, with a fresh commit)
+  // ===========================================================================
+  it should "leave committed objects intact while resetting expired staged objects" in {
+    // Commit a new object in this test, alongside an expired staged object.
+    val committedPath = uniquePath("committed-safe")
+    stageObject(committedPath, content = "committed-bytes")
+    LakeFSStorageClient.createCommit(repoName, "main", "commit object that cleanup must keep")
+
+    val expiredStaged = uniquePath("doomed-staged")
+    stageObject(expiredStaged)
+
+    val report = job.runCleanupOnce(farFuture)
+
+    report.errors shouldEqual 0
+    uncommittedPaths() should not contain expiredStaged
+    committedPaths() should contain(committedPath)
+    committedPaths() should contain(PinnedCommittedPath)
+  }
+
+  // ===========================================================================
+  // 10. Already-aborted multipart (LakeFS 404) is treated as cleaned, not an error
+  // ===========================================================================
+  it should "delete a session whose multipart was already aborted in LakeFS, with no error" in {
+    val filePath = uniquePath("already-aborted")
+    initSession(filePath)
+    val record = fetchSession(filePath)
+    // Abort the multipart out-of-band; the DB row stays behind (simulates a crash between
+    // LakeFS abort and row deletion, or a previous partially-failed cleanup round).
+    LakeFSStorageClient.abortPresignedMultipartUploads(
+      repoName,
+      filePath,
+      record.getUploadId,
+      record.getPhysicalAddress
+    )
+    fetchSession(filePath) should not be null
+
+    val report = job.runCleanupOnce(farFuture)
+
+    report.sessionsDeleted shouldEqual 1
+    report.errors shouldEqual 0
+    fetchSession(filePath) shouldBe null
+  }
+
+  // ===========================================================================
+  // 11. Per-item failures are counted and never abort the batch
+  // ===========================================================================

Review Comment:
   let's also test for different step failure during clean one item, such as no DB record is found, no file found, and timeout, such that the resulting state can either be rolled back, or cleaned in next round. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3483089015


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList
+
+    expiredSessions.foreach { session =>
+      try {
+        // Delete the row and abort the multipart in one transaction, deleting FIRST. LakeFS is
+        // external and cannot truly enroll in a DB transaction, but the abort is idempotent
+        // (re-aborting an already-aborted upload returns 404, treated as success below), so the
+        // only risk is the abort failing AFTER the delete is staged. By staging the delete first
+        // and letting a non-404 abort failure roll the whole transaction back, the session row
+        // survives and the next round retries — never leaving an orphaned multipart behind.
+        SqlServer.withTransaction(ctx) { txn =>
+          txn
+            .deleteFrom(DATASET_UPLOAD_SESSION)
+            .where(DATASET_UPLOAD_SESSION.UPLOAD_ID.eq(session.getUploadId))
+            .execute()
+          repoNameByDid.get(session.getDid) match {
+            case Some(repoName) =>
+              try {
+                LakeFSStorageClient.abortPresignedMultipartUploads(
+                  repoName,
+                  session.getFilePath,
+                  session.getUploadId,
+                  session.getPhysicalAddress
+                )
+              } catch {
+                // Already aborted (or never materialized): safe to delete the session row.
+                case e: ApiException if e.getCode == 404 =>
+                  logger.debug(
+                    s"Multipart upload ${session.getUploadId} not found in LakeFS; " +
+                      "treating as already aborted"
+                  )
+              }
+            case None =>
+              // Dataset row gone or repository_name is NULL: the multipart lived in that
+              // repository's namespace, so there is nothing left to abort.
+              logger.debug(
+                s"No repository for dataset ${session.getDid}; " +
+                  s"deleting orphan upload session ${session.getUploadId}"
+              )
+          }
+        }
+        sessionsDeleted += 1
+      } catch {
+        case t: Throwable =>
+          logger.warn(
+            s"Failed to clean up upload session ${session.getUploadId} " +
+              s"(did=${session.getDid}, path=${session.getFilePath}); will retry next round",
+            t
+          )
+          errors += 1
+      }
+    }
+
+    // File paths of still-active (non-expired) upload sessions, per dataset. Staged objects
+    // belonging to an active upload must never be reset.
+    val activePathsByDid: Map[Integer, Set[String]] = ctx
+      .select(DATASET_UPLOAD_SESSION.DID, DATASET_UPLOAD_SESSION.FILE_PATH)
+      .from(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.ge(cutoff))
+      .fetch()
+      .asScala
+      .groupBy(record => record.get(DATASET_UPLOAD_SESSION.DID))
+      .map {
+        case (did, records) =>
+          did -> records.map(_.get(DATASET_UPLOAD_SESSION.FILE_PATH)).toSet
+      }
+
+    // Path 2: reset staged (uncommitted) objects older than the retention window.
+    repoNameByDid.foreach {
+      case (did, repoName) =>
+        try {
+          val activePaths = activePathsByDid.getOrElse(did, Set.empty)
+          val stagedObjects = LakeFSStorageClient.retrieveUncommittedObjects(repoName)
+          // diffBranch carries no mtime, so each candidate costs one extra statObject call
+          // (N+1). Unavoidable until LakeFS exposes timestamps in the diff API.
+          stagedObjects.foreach { diff =>
+            val path = diff.getPath
+            val isObjectWrite =
+              diff.getType == Diff.TypeEnum.ADDED || diff.getType == Diff.TypeEnum.CHANGED
+            if (!isObjectWrite) {
+              // E.g. a staged deletion of a committed file: there is no object behind it and
+              // it consumes no storage, so leaving it is correct and cheap.
+              logger.debug(s"Skipping staged ${diff.getType} entry '$path' in '$repoName'")
+            } else if (!activePaths.contains(path)) {
+              try {
+                val mtime = LakeFSStorageClient.getStagedObjectMtime(repoName, path)
+                if (mtime < cutoff.toEpochSecond) {
+                  LakeFSStorageClient.resetObjectUploadOrDeletion(repoName, path)
+                  objectsReset += 1
+                }
+              } catch {
+                // Concurrently committed/reset, or already cleaned by another round: the
+                // object is gone, which is the desired end state for an idempotent job.
+                case e: ApiException if e.getCode == 404 =>
+                  logger.debug(
+                    s"Staged object '$path' not found in repo '$repoName'; " +
+                      "treating as already cleaned"
+                  )
+                case t: Throwable =>
+                  logger.warn(
+                    s"Failed to clean up staged object '$path' in repo '$repoName'",
+                    t
+                  )
+                  errors += 1
+              }
+            }
+          }
+        } catch {
+          // The dataset's LakeFS repository was deleted out-of-band (a supported state):
+          // nothing staged to clean up there.
+          case e: ApiException if e.getCode == 404 =>
+            logger.debug(s"Repository '$repoName' not found in LakeFS; skipping")
+          case t: Throwable =>
+            logger.warn(s"Failed to clean up staged objects in repo '$repoName'", t)
+            errors += 1
+        }
+    }
+
+    logger.info(

Review Comment:
   Resolved. Per-round summary is now at debug level



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on code in PR #5643:
URL: https://github.com/apache/texera/pull/5643#discussion_r3483079065


##########
file-service/src/main/scala/org/apache/texera/service/util/StagedFileCleanupJob.scala:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.dropwizard.lifecycle.Managed
+import io.lakefs.clients.sdk.ApiException
+import io.lakefs.clients.sdk.model.Diff
+import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.Dataset.DATASET
+import org.apache.texera.dao.jooq.generated.tables.DatasetUploadSession.DATASET_UPLOAD_SESSION
+
+import java.time.OffsetDateTime
+import java.util.concurrent.{Executors, ScheduledExecutorService, TimeUnit}
+import scala.jdk.CollectionConverters._
+
+/**
+  * Summary of one cleanup round.
+  *
+  * @param sessionsDeleted Number of abandoned upload session rows deleted.
+  * @param objectsReset    Number of staged (uncommitted) objects reset in LakeFS.
+  * @param errors          Number of failures encountered (each is retried next round).
+  */
+case class CleanupReport(sessionsDeleted: Int, objectsReset: Int, errors: Int)
+
+/**
+  * Periodically cleans up uploaded but uncommitted dataset files:
+  *   1. Aborts and deletes abandoned multipart upload sessions older than the retention window.
+  *   2. Resets staged (uncommitted) LakeFS objects older than the retention window, skipping
+  *      objects that belong to still-active upload sessions.
+  *
+  * @param retentionHours  Age (in hours) after which uncommitted uploads are cleaned up.
+  * @param intervalMinutes Delay (in minutes) between cleanup rounds.
+  */
+class StagedFileCleanupJob(retentionHours: Int, intervalMinutes: Int)
+    extends Managed
+    with LazyLogging {
+
+  require(retentionHours > 0, s"retentionHours must be > 0 (got $retentionHours)")
+  require(intervalMinutes > 0, s"intervalMinutes must be > 0 (got $intervalMinutes)")
+
+  private var executor: ScheduledExecutorService = _
+
+  override def start(): Unit = {
+    executor = Executors.newSingleThreadScheduledExecutor((runnable: Runnable) => {
+      val thread = new Thread(runnable, "staged-file-cleanup")
+      thread.setDaemon(true)
+      thread
+    })
+    executor.scheduleWithFixedDelay(
+      () => {
+        try {
+          runCleanupOnce()
+        } catch {
+          // An exception must never kill the schedule.
+          case t: Throwable => logger.error("Staged file cleanup round failed", t)
+        }
+      },
+      // Small fixed initial delay so a restart doesn't postpone backlog cleanup by up to a
+      // full interval.
+      1L,
+      intervalMinutes.toLong,
+      TimeUnit.MINUTES
+    )
+  }
+
+  override def stop(): Unit = {
+    if (executor != null) {
+      executor.shutdown()
+    }
+  }
+
+  /**
+    * Runs a single cleanup round. Idempotent: rows/objects already cleaned up are not
+    * revisited, and failures are retried on the next round.
+    *
+    * @param now The reference time used to evaluate the retention window.
+    * @return Summary counts for this round.
+    */
+  def runCleanupOnce(now: OffsetDateTime = OffsetDateTime.now()): CleanupReport = {
+    val cutoff = now.minusHours(retentionHours.toLong)
+    var sessionsDeleted = 0
+    var objectsReset = 0
+    var errors = 0
+
+    val ctx = SqlServer.getInstance().createDSLContext()
+
+    // Map each dataset id to its LakeFS repository name (same mapping DatasetResource uses
+    // via dataset.getRepositoryName).
+    val repoNameByDid: Map[Integer, String] = ctx
+      .select(DATASET.DID, DATASET.REPOSITORY_NAME)
+      .from(DATASET)
+      .where(DATASET.REPOSITORY_NAME.isNotNull)
+      .fetch()
+      .asScala
+      .map(record => record.get(DATASET.DID) -> record.get(DATASET.REPOSITORY_NAME))
+      .toMap
+
+    // Path 1: abort and delete abandoned multipart upload sessions.
+    val expiredSessions = ctx
+      .selectFrom(DATASET_UPLOAD_SESSION)
+      .where(DATASET_UPLOAD_SESSION.CREATED_AT.lt(cutoff))
+      .fetch()
+      .asScala
+      .toList

Review Comment:
   Resolved. The session fetch is bounded with .orderBy(created_at).limit(500)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(file-service): cleanup stale uncommitted dataset uploads [texera]

Posted by "eugenegujing (via GitHub)" <gi...@apache.org>.
eugenegujing commented on PR #5643:
URL: https://github.com/apache/texera/pull/5643#issuecomment-4783874447

   @Yicong-Huang Thanks for the review! Addressed all the comments: storage.cleanup.enabled now defaults to false, runCleanupOnce is private[util], the per-round summary is now at debug level, and the session fetch is bounded with .orderBy(created_at).limit(500) so it can't OOM on a large table. The path-2 race is fixed by dropping the stale activePathsByDid snapshot in favor of a live hasActiveUploadSession check plus an mtime re-read right before each reset; the only remaining window is the irreducible sub-microsecond gap before the reset, since LakeFS can't join the DB transaction. On coverage, I expanded StagedFileCleanupJobSpec to 21 cases and added StorageConfigSpec and a LakeFSStorageClientSpec backed by real Postgres+MinIO+LakeFS testcontainers; the only lines left uncovered are the FileService start-up wiring and the scheduler/race-guard branch. Could you please review it again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org