You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rdblue (via GitHub)" <gi...@apache.org> on 2026/05/20 21:34:35 UTC

[I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

rdblue opened a new issue, #16493:
URL: https://github.com/apache/iceberg/issues/16493

   > This issue was reported to the private Apache Iceberg security mailing list. The submitter is being kept anonymous because the report was sent to a private list. After review, the issue is not considered a serious vulnerability that needs to be kept private, so it is being filed publicly here for tracking and resolution.
   >
   > Note: this submission was generated by AI. Please review its claims and source references carefully before acting on them.
   
   # Summary
   
   Spark scopes `file_list_view` with raw string prefix matching, letting
   sibling paths fall inside orphan cleanup.
   
   # Affected Maven coordinates
   
   * versioned integration artifacts:
   `org.apache.iceberg:iceberg-spark-3.4_*`,
   `org.apache.iceberg:iceberg-spark-3.5_*`,
   `org.apache.iceberg:iceberg-spark-4.0_2.13`,
   `org.apache.iceberg:iceberg-spark-4.1_2.13`
   
   # Attacker prerequisites
   
   * control over a path, prefix, batch, or file list that sits adjacent
   to a legitimately scoped prefix
   * ability to trigger the credential-selection or cleanup path with
   that crafted input
   
   # Impact
   
   * A table location like `s3://bucket/table` also matches sibling
   prefixes such as `s3://bucket/table-backup/...`.
   * If the caller supplies a crafted `file_list_view`, files outside the
   intended directory can be treated as in-scope and deleted.
   * This weakens scope checks even when the explicit `location` override
   is not used.
   
   # Proof status
   
   Source review only. The issue is visible directly from source.
   
   # Key source references
   
   * org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction
   
   Current severity assessment [2]: Moderate
   
   [1] https://iceberg.apache.org/security/
   [2] https://security.apache.org/blog/severityrating/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "CoderNasim2023 (via GitHub)" <gi...@apache.org>.
CoderNasim2023 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-5602599895

   Hey I just Create a PR and solved the issue please take a look    
   Spark: Fix orphan file sibling path matching #18029 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "neon-hippo (via GitHub)" <gi...@apache.org>.
neon-hippo commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4515504078

   Before submitting a fix, I wanted to raise a design question. The same problem — "is this file path under the table location?" — is solved two different ways:
   
    - listedFileDS() uses filesystem APIs (Hadoop listStatus, FileSystemWalker) which natively respect directory boundaries
    - filteredCompareToFileList() uses raw String.startsWith(location) which doesn't
   
   Is there a reason filteredCompareToFileList() can't use the same filesystem-based approach instead of string matching? That would remove the bug entirely and reduce the two implementations to one. If there's a deliberate reason for the string-based path (performance on S3 file lists?), I'd like to understand it before attempting a fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4684422937

   > One follow-up: I noticed `filteredCompareToFileList()` line 228 also uses a raw `startsWith(location)` without path separator validation. While this doesn't make storage calls, it could still match sibling paths if the caller's file list contains them (e.g., `table-backup/` matching `table`).
   
   It seems reasonable to fix this in a separate PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4503727765

   Not a security issue. Anyone who can run remove_orphan_files with a custom file_list_view is already trusted to do destructive maintenance on that table’s storage. The startsWith filter is sloppy and worth fixing as a normal bug, but since a caller can always just specify "location" to the target prefix and get the same behavior.
   
   So we can keep this open, but i'll remove the security tag
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4557915102

   @nnguyen168, I don't understand your question.
   
   The method `filteredCompareToFileList` returns a `Dataset` that was provided by the caller after adding a filter to exclude recently modified files. Because this was provided by the caller, Iceberg is not using a credential to get the paths and it is not going to fail as described.
   
   The method `listedFileDS` uses either `FileSystemWalker.listDirRecursivelyWithFileIO` or `FileSystemWalker.listDirRecursivelyWithHadoop` to get a list of files in storage. This is what needs to be updated to avoid listing paths without `/`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4510881545

   I talked about this one with @danielcweeks and @nastra and it's probably a good idea to look for cases like this and ensure that we are only creating requests with the path separator. It's not that this is a security issue (it's the service's job to vend an appropriately scoped credential) but we don't want clients to fail if they attempt to list `s3://bucket/table` and don't have permission.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "nnguyen168 (via GitHub)" <gi...@apache.org>.
nnguyen168 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4516458279

   Hi, I'd like to work on this issue. Before I start, could a committer clarify the preferred approach?
   
     1. Quick fix: Add path separator check to the startsWith comparison in filteredCompareToFileList()
     2. Broader fix: Unify both listedFileDS() and filteredCompareToFileList() to use the same path-based approach
   
   @rdblue @RussellSpitzer  which direction would you prefer?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "CoderNasim2023 (via GitHub)" <gi...@apache.org>.
CoderNasim2023 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4829280442

   Hey  I wants to solve this ASAP  , please give me  a  oppourtunity to prove 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "CoderNasim2023 (via GitHub)" <gi...@apache.org>.
CoderNasim2023 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-6144825140

   > 16\. Hey [@rdblue](https://github.com/rdblue)  I just Created a PR and solved the issue please take a look
   >      Spark: Fix orphan file sibling path matching [#18029](https://github.com/apache/iceberg/pull/18029)
   
     what  about  the latest updates  of this 2 pr  #18029 #16498  @rdblue  @wombatu-kun  ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "txwyy123 (via GitHub) via issues" <is...@iceberg.apache.org>.
txwyy123 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-6650945795

   This is addressed for the `filteredCompareToFileList` path in #18768 — the separate fix that @rdblue carved out in [this comment](https://github.com/apache/iceberg/issues/16493#issuecomment-4557915102). The `startsWith(location)` filter now requires a path separator after the location prefix, preventing sibling paths from being incorrectly matched. CI is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "maoli67660 (via GitHub)" <gi...@apache.org>.
maoli67660 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4880034907

   Hi, I'd like to work on this issue. I've identified the bug in `filteredCompareToFileList()` in `DeleteOrphanFilesSparkAction.java `where `startsWith(location)` without a trailing / can incorrectly match sibling paths. 
   
   I'll submit a fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `remove_orphan_files` scopes `file_list_view` with raw string prefix matching [iceberg]

Posted by "nnguyen168 (via GitHub)" <gi...@apache.org>.
nnguyen168 commented on issue #16493:
URL: https://github.com/apache/iceberg/issues/16493#issuecomment-4563398490

   Thanks @rdblue for the clarification. I understand now that the primary concern is `listedFileDS()` and the `FileSystemWalker` methods making storage requests without the trailing `/`.
   
   One follow-up: I noticed `filteredCompareToFileList()` line 228 also uses a raw `startsWith(location)` without path separator validation. While this doesn't make storage calls, it could still match sibling paths if the caller's file list contains them (e.g., `table-backup/` matching `table`).
   
   Should this also be fixed as part of this issue, or is that considered acceptable since the caller provided the file list?
   
   I'll focus on `FileSystemWalker.listDirRecursivelyWithFileIO` and `listDirRecursivelyWithHadoop` for the main fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org