You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "FrankChen021 (via GitHub)" <gi...@apache.org> on 2026/08/17 07:54:59 UTC
[PR] fix(hdfs): remove LZ4 segments during kill tasks (druid)
FrankChen021 opened a new pull request, #20029:
URL: https://github.com/apache/druid/pull/20029
## Description
Allow the HDFS segment killer to delete segments stored with the LZ4 compression format.
## Root cause
HDFS LZ4 support writes segment files with an `index.lz4` suffix, but `HdfsDataSegmentKiller` only accepted `.zip` files and validated `index.zip` in partitioned filenames. Kill tasks could remove the segment metadata and then fail before deleting the LZ4 object from deep storage.
## Changes
- Accept ZIP and LZ4 segment formats in `HdfsDataSegmentKiller`.
- Validate partitioned filenames against the detected compression suffix.
- Add a regression test that creates and removes a `3_index.lz4` segment and verifies empty parent directories are cleaned up.
## Tests
- `env JAVA_HOME=/Library/Java/JavaVirtualMachines/temurin-25.jdk/Contents/Home mvn test -pl extensions-core/hdfs-storage -am -Dtest='org.apache.druid.storage.hdfs.HdfsDataSegmentKillerTest' -Dsurefire.failIfNoSpecifiedTests=false -Pskip-static-checks -Dweb.console.skip=true -T1C`
- 11 tests passed
- [x] Self-reviewed
- [x] Added unit test coverage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix(hdfs): remove LZ4 segments during kill tasks (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 merged PR #20029:
URL: https://github.com/apache/druid/pull/20029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org