You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2026/07/06 10:49:50 UTC
[PR] feat: add FileNameParser for file name decoding [hudi]
danny0405 opened a new pull request, #19198:
URL: https://github.com/apache/hudi/pull/19198
### Describe the issue this Pull Request addresses
This closes: #19197
Hudi file-name decoding is currently spread across `FSUtils`, `HoodieBaseFile`, `HoodieLogFile`, and several read/write call sites using repeated regex matching and ad hoc string splitting. This makes the behavior harder to reason about, especially now that both inline log files and native log files have to be supported consistently.
This PR centralizes Hudi-generated base/log file-name parsing into a dedicated parser and routes existing helpers through that single decoding path. It also tightens base-file parsing so base files without a file extension are rejected, matching the reader-side expectation that supported base files have a concrete format extension.
This PR changes common helper APIs: file-id prefix parsing moves from `FSUtils.getFileIdPfxFromFileId` to `FileNameParser.getFileIdPfxFromFileId`, and inline log file name construction is made explicit via `FSUtils.makeInlineLogFileName`. There is no storage format change.
### Summary and Changelog
File-name parsing for Hudi base files, inline log files, and native log files is now centralized in `FileNameParser`, with `FSUtils`, `HoodieBaseFile`, and `HoodieLogFile` delegating to the decoded parser result instead of duplicating regex logic.
#### Working tree: Centralize Hudi file-name parsing
- Added `FileNameParser` for decoding base file names, inline log file names, native log file names, CDC log files, and file-id prefixes.
- Removed base/log/native regex ownership from `FSUtils` and routed filename detail helpers through `FileNameParser`.
- Removed fallback parsing for extensionless base files; `HoodieBaseFile` now throws `InvalidHoodiePathException` for unsupported Hudi-generated base file names without an extension.
- Updated `HoodieLogFile` to decode log metadata through `FileNameParser`, including native-log detection and CDC detection.
- Renamed inline log file construction to `FSUtils.makeInlineLogFileName` to distinguish it from native log file naming.
- Moved `getFileIdPfxFromFileId` usage to `FileNameParser` in `ConsistentBucketIndexUtils` and `RDDSimpleBucketBulkInsertPartitioner`.
- Reduced repeated file-name parsing in Spark/Flink reader contexts, CDC split construction, and log-reader summary logging.
- Updated tests and fixtures to use supported base file names with extensions and the explicit inline log file-name helper.
- Added parser coverage in `TestFSUtils` for base files, inline log files, native log files, CDC suffix handling, and file-id prefix validation.
- Added `TestHoodieBaseFile` coverage for rejecting base files without extensions.
### Impact
This affects common file-name parsing paths used by file-system views, base/log file models, Spark/Flink reader contexts, CDC utilities, and bucket-index helpers.
There is no table format or storage layout change. Existing valid Hudi base files and log files should continue to parse the same way. Invalid extensionless base file names are now rejected instead of being partially decoded.
Public helper API impact:
- `FSUtils.getFileIdPfxFromFileId` is removed in favor of `FileNameParser.getFileIdPfxFromFileId`.
- Inline log file construction is now named `FSUtils.makeInlineLogFileName`.
### Risk Level
medium
The risk is medium because this touches core filename parsing helpers used across read paths, write paths, file-system views, and tests. The risk is mitigated by centralizing behavior in one parser, preserving existing decoded field semantics, and adding focused parser and model tests.
Validated with:
- `git diff --check`
- `mvn -pl hudi-hadoop-common -am -DskipITs -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dtest=TestFSUtils test`
- `mvn -pl hudi-hadoop-common -am -DskipITs -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dtest=TestHoodieLogFile,TestFSUtils,TestHoodieLogFormatWriterBuilder test`
### Documentation Update
none
This is a common-code refactor and helper API cleanup with inline Javadocs added to `FileNameParser`. No user-facing configuration, table service behavior, or storage format documentation is required.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4893753983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8172034c4fe29ac4351e3fce3dda6e7af951f1d4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263)
* e186f84ac1b8dbd335715860808b92d7810c6e9b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3528965858
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java:
##########
@@ -519,54 +452,31 @@ public static boolean isLogFile(StoragePath logPath) {
}
public static boolean isLogFile(String fileName) {
- if (matchNativeLogFile(fileName).isPresent()) {
- return true;
- }
- if (fileName.startsWith(LOG_FILE_START_WITH_CHARACTER)) {
- Matcher matcher = LOG_FILE_PATTERN.matcher(fileName);
- return matcher.matches() && matcher.group(3).equals(LOG_FILE_EXTENSION);
- }
- return false;
+ return FileNameParser.parseLogFile(fileName)
+ .map(logFileName -> logFileName.isNativeLogFile()
+ || logFileName.getFileExtension().equals(LogExtensions.DATA_LOG_EXTENSION))
+ .orElse(false);
}
public static boolean isInlineLogFile(String fileName) {
- return matchNativeLogFile(fileName).isEmpty();
+ return FileNameParser.parseNativeLogFile(fileName).isEmpty();
}
public static Option<Matcher> matchNativeLogFile(String fileName) {
- if (StringUtils.isNullOrEmpty(fileName)) {
- return Option.empty();
- }
- String actualFileName = fileName.contains(StoragePath.SEPARATOR)
- ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
- : fileName;
- Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
- return matcher.matches() ? Option.of(matcher) : Option.empty();
+ return FileNameParser.matchNativeLogFile(fileName);
Review Comment:
Since the goal is to centralize name decoding, can we also migrate the remaining `Option<Matcher>` consumers to the typed API? `NativeLogFooterMetadata` still reads `group(8)`/`group(9)` and `TableSchemaResolver` passes the raw `Matcher` through, while `HoodieLogFormat`/`HoodieLogFormatReader`/`CdcIterators`/`CDCFileGroupIterator` only need an `isNativeLogFile` boolean. Keeping the `Matcher`-returning API public means a future regex/group-index change would break these callers silently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4904207781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323)
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4909544617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4904332331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323)
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4904776533
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323)
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4900129539
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4910750926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
}, {
"hash" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352)
* 4fca0f381d302adecfd56e4bf76937cc84e06117 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4910916532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
}, {
"hash" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"triggerType" : "PUSH"
}, {
"hash" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"triggerType" : "PUSH"
}, {
"hash" : "bd2283fac7888c22734c75858caf83c3fd53e6d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd2283fac7888c22734c75858caf83c3fd53e6d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352)
* 4fca0f381d302adecfd56e4bf76937cc84e06117 UNKNOWN
* 31be9fcdb2d3389d720738b9202c6d34421ba619 UNKNOWN
* bd2283fac7888c22734c75858caf83c3fd53e6d0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899379112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4900027632
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3529462461
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi-generated base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ static final Pattern BASE_FILE_PATTERN = Pattern.compile("[a-zA-Z0-9-]+_[a-zA-Z0-9-]+_[0-9]+\\.[a-zA-Z0-9]+");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a Hudi-generated base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>.<extension>}.
+ * The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot, matching existing
+ * base-file helper API behavior.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches the Hudi base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Matcher matcher = BASE_FILE_PATTERN.matcher(actualFileName);
+ if (!matcher.matches()) {
+ return Option.empty();
+ }
+
+ int firstUnderscoreIndex = actualFileName.indexOf('_');
+ int secondUnderscoreIndex = actualFileName.indexOf('_', firstUnderscoreIndex + 1);
+ int dotIndex = actualFileName.indexOf('.', secondUnderscoreIndex + 1);
+ return Option.of(new BaseFileName(
+ actualFileName.substring(0, firstUnderscoreIndex),
+ actualFileName.substring(firstUnderscoreIndex + 1, secondUnderscoreIndex),
+ actualFileName.substring(secondUnderscoreIndex + 1, dotIndex),
+ actualFileName.substring(dotIndex)));
+ }
+
+ /**
+ * Parses either an inline log file name or a native log file name.
+ *
+ * <p>Inline log format: {@code .<fileId>_<deltaCommitTime>.<extension>.<version>_<writeToken>[.cdc]}.
+ * Native log format: {@code <fileId>_<writeToken>_<deltaCommitTime>_<version>.<extension>.<nativeFormat>}.
+ * The decoded {@link LogFileName#getFileExtension()} value does not include a leading dot. For inline CDC
+ * logs, {@link LogFileName#getSuffix()} is {@code .cdc}; for native logs, it is the native storage format,
+ * such as {@code parquet}.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded log file name when the input matches either supported log-file format
+ */
+ public static Option<LogFileName> parseLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Option<LogFileName> nativeLogFileName = parseNativeLogFileFromActualFileName(actualFileName);
+ if (nativeLogFileName.isPresent()) {
+ return nativeLogFileName;
+ }
+
+ Matcher matcher = INLINE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(LogFileName.fromInlineLogFile(matcher)) : Option.empty();
+ }
+
+ /**
+ * Parses only a native log file name.
+ *
+ * <p>Use {@link #parseLogFile(String)} when both inline and native log formats are acceptable.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded native log file name when the input matches the native log-file format
+ */
+ public static Option<LogFileName> parseNativeLogFile(String fileName) {
+ return matchNativeLogFile(fileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ /**
+ * Returns the file group prefix encoded in a Hudi file id.
+ *
+ * @param fileId Hudi file id ending with a numeric suffix separated by {@code -}
+ * @return file id prefix before the trailing numeric suffix
+ * @throws HoodieValidationException when the file id does not contain the expected suffix
+ */
+ public static String getFileIdPfxFromFileId(String fileId) {
+ Matcher matcher = PREFIX_BY_FILE_ID_PATTERN.matcher(fileId);
+ if (!matcher.find()) {
+ throw new HoodieValidationException("Failed to get prefix from " + fileId);
+ }
+ return matcher.group(1);
+ }
+
+ static Option<Matcher> matchNativeLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ return matchNativeLogFileFromActualFileName(actualFileName);
+ }
+
+ private static Option<LogFileName> parseNativeLogFileFromActualFileName(String actualFileName) {
+ return matchNativeLogFileFromActualFileName(actualFileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ private static Option<Matcher> matchNativeLogFileFromActualFileName(String actualFileName) {
+ Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(matcher) : Option.empty();
+ }
+
+ private static String getActualFileName(String fileName) {
+ return fileName.contains(StoragePath.SEPARATOR)
+ ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
+ : fileName;
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated base file name.
+ */
+ @Getter
+ public static final class BaseFileName {
+ private final String fileId;
+ private final String writeToken;
+ private final String commitTime;
+ private final String fileExtension;
+
+ private BaseFileName(String fileId, String writeToken, String commitTime, String fileExtension) {
+ this.fileId = fileId;
+ this.writeToken = writeToken;
+ this.commitTime = commitTime;
+ this.fileExtension = fileExtension;
+ }
+
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated log file name.
+ */
+ @Getter
+ public static final class LogFileName {
+ private final String fileId;
+ private final String deltaCommitTime;
+ private final int logVersion;
+ private final String writeToken;
+ private final Integer taskPartitionId;
+ private final Integer stageId;
+ private final Integer taskAttemptId;
+ private final String fileExtension;
+ private final String suffix;
Review Comment:
Done. I kept the public/accessor name as suffix, but added a declaration-site comment explaining that inline logs use it for the optional .cdc suffix while native logs use it for the native storage format such as parquet.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieLogFile.java:
##########
@@ -181,7 +173,9 @@ public String getFileExtension() {
}
public boolean isCDC() {
- return FSUtils.isCDCLogFile(getFileName());
+ return isNativeLogFile()
Review Comment:
Done. HoodieLogFile now caches the CDC classification from FileNameParser.LogFileName#isCDCLogFile during parseFieldsFromPath, so the CDC predicate logic stays centralized in FileNameParser.
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java:
##########
@@ -519,54 +452,31 @@ public static boolean isLogFile(StoragePath logPath) {
}
public static boolean isLogFile(String fileName) {
- if (matchNativeLogFile(fileName).isPresent()) {
- return true;
- }
- if (fileName.startsWith(LOG_FILE_START_WITH_CHARACTER)) {
- Matcher matcher = LOG_FILE_PATTERN.matcher(fileName);
- return matcher.matches() && matcher.group(3).equals(LOG_FILE_EXTENSION);
- }
- return false;
+ return FileNameParser.parseLogFile(fileName)
+ .map(logFileName -> logFileName.isNativeLogFile()
+ || logFileName.getFileExtension().equals(LogExtensions.DATA_LOG_EXTENSION))
+ .orElse(false);
}
public static boolean isInlineLogFile(String fileName) {
- return matchNativeLogFile(fileName).isEmpty();
+ return FileNameParser.parseNativeLogFile(fileName).isEmpty();
}
public static Option<Matcher> matchNativeLogFile(String fileName) {
- if (StringUtils.isNullOrEmpty(fileName)) {
- return Option.empty();
- }
- String actualFileName = fileName.contains(StoragePath.SEPARATOR)
- ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
- : fileName;
- Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
- return matcher.matches() ? Option.of(matcher) : Option.empty();
+ return FileNameParser.matchNativeLogFile(fileName);
Review Comment:
Done. The remaining raw Matcher consumers were migrated: TableSchemaResolver and NativeLogFooterMetadata now use FileNameParser.LogFileName, and the read/CDC paths use FSUtils.isNativeLogFile. FSUtils.matchNativeLogFile has been removed from the public API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899492260
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4900157275
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3528967044
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieLogFile.java:
##########
@@ -181,7 +173,9 @@ public String getFileExtension() {
}
public boolean isCDC() {
- return FSUtils.isCDCLogFile(getFileName());
+ return isNativeLogFile()
Review Comment:
This inlines the same ternary as `FileNameParser.LogFileName#isCDCLogFile()`, so CDC detection now lives in two places that must stay in sync. Since `parseFieldsFromPath()` already has the parsed `LogFileName`, could we cache the boolean there (or delegate to a shared static predicate)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4893807109
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8172034c4fe29ac4351e3fce3dda6e7af951f1d4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263)
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4892031120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8172034c4fe29ac4351e3fce3dda6e7af951f1d4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899822868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4901569935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3533555023
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.ExternalFilePathUtil;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+ private static final char UNDERSCORE = '_';
+ private static final char DOT = '.';
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>[.<extension>]}.
+ * Decoding mirrors {@code HoodieBaseFile}'s historical substring scan: the first underscore terminates
+ * the file id, the second underscore precedes the commit time, and the first dot after the second
+ * underscore terminates the commit time. If the dot is absent, the commit time extends to the end of
+ * the file name. The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot when
+ * an extension is present.</p>
+ *
+ * <p>Externally registered base files marked with {@code _hudiext} are decoded through
+ * {@link ExternalFilePathUtil}. For these files, {@link BaseFileName#getWriteToken()} is empty and
+ * {@link BaseFileName#getFileExtension()} is derived from the original external file name.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches a supported base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ if (ExternalFilePathUtil.isExternallyCreatedFile(actualFileName)) {
+ return parseExternalBaseFile(actualFileName);
+ }
+
+ int firstUnderscoreIndex = -1;
+ int secondUnderscoreIndex = -1;
+ int dotIndex = -1;
+ short underscoreCount = 0;
+ for (int i = 0; i < actualFileName.length(); i++) {
+ char c = actualFileName.charAt(i);
+ if (c == UNDERSCORE) {
+ if (underscoreCount == 0) {
+ firstUnderscoreIndex = i;
+ } else if (underscoreCount == 1) {
+ secondUnderscoreIndex = i;
+ }
+ underscoreCount++;
+ } else if (c == DOT && underscoreCount == 2) {
+ dotIndex = i;
+ break;
+ }
+ }
+ if (firstUnderscoreIndex < 0 || secondUnderscoreIndex < 0) {
+ return Option.empty();
+ }
+ int commitTimeEndIndex = dotIndex < 0 ? actualFileName.length() : dotIndex;
+ return Option.of(new BaseFileName(
+ actualFileName.substring(0, firstUnderscoreIndex),
+ actualFileName.substring(firstUnderscoreIndex + 1, secondUnderscoreIndex),
+ actualFileName.substring(secondUnderscoreIndex + 1, commitTimeEndIndex),
+ dotIndex < 0 ? "" : actualFileName.substring(dotIndex)));
+ }
+
+ /**
+ * Parses either an inline log file name or a native log file name.
+ *
+ * <p>Inline log format: {@code .<fileId>_<deltaCommitTime>.<extension>.<version>_<writeToken>[.cdc]}.
+ * Native log format: {@code <fileId>_<writeToken>_<deltaCommitTime>_<version>.<extension>.<nativeFormat>}.
+ * The decoded {@link LogFileName#getFileExtension()} value does not include a leading dot. For inline CDC
+ * logs, {@link LogFileName#getSuffix()} is {@code .cdc}; for native logs, it is the native storage format,
+ * such as {@code parquet}.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded log file name when the input matches either supported log-file format
+ */
+ public static Option<LogFileName> parseLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Option<LogFileName> nativeLogFileName = parseNativeLogFileFromActualFileName(actualFileName);
+ if (nativeLogFileName.isPresent()) {
+ return nativeLogFileName;
+ }
+
+ Matcher matcher = INLINE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(LogFileName.fromInlineLogFile(matcher)) : Option.empty();
+ }
+
+ /**
+ * Parses only a native log file name.
+ *
+ * <p>Use {@link #parseLogFile(String)} when both inline and native log formats are acceptable.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded native log file name when the input matches the native log-file format
+ */
+ public static Option<LogFileName> parseNativeLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ return parseNativeLogFileFromActualFileName(actualFileName);
+ }
+
+ /**
+ * Returns the file group prefix encoded in a Hudi file id.
+ *
+ * @param fileId Hudi file id ending with a numeric suffix separated by {@code -}
+ * @return file id prefix before the trailing numeric suffix
+ * @throws HoodieValidationException when the file id does not contain the expected suffix
+ */
+ public static String getFileIdPfxFromFileId(String fileId) {
+ Matcher matcher = PREFIX_BY_FILE_ID_PATTERN.matcher(fileId);
+ if (!matcher.find()) {
+ throw new HoodieValidationException("Failed to get prefix from " + fileId);
+ }
+ return matcher.group(1);
+ }
+
+ private static Option<LogFileName> parseNativeLogFileFromActualFileName(String actualFileName) {
+ return matchNativeLogFileFromActualFileName(actualFileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ private static Option<Matcher> matchNativeLogFileFromActualFileName(String actualFileName) {
+ Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(matcher) : Option.empty();
+ }
+
+ private static String getActualFileName(String fileName) {
+ return fileName.contains(StoragePath.SEPARATOR)
+ ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
+ : fileName;
+ }
+
+ private static Option<BaseFileName> parseExternalBaseFile(String actualFileName) {
+ String[] fileIdAndCommitTime = ExternalFilePathUtil.parseFileIdAndCommitTimeFromExternalFile(actualFileName);
+ return Option.of(new BaseFileName(fileIdAndCommitTime[0], "", fileIdAndCommitTime[1],
+ getFileExtension(fileIdAndCommitTime[0])));
+ }
+
+ private static String getFileExtension(String fileName) {
+ int lastSeparatorIndex = fileName.lastIndexOf(StoragePath.SEPARATOR);
+ int dotIndex = fileName.lastIndexOf(DOT);
+ return dotIndex > lastSeparatorIndex ? fileName.substring(dotIndex) : "";
+ }
+
+ /**
+ * Decoded parts of a base file name.
+ */
+ @Getter
+ public static final class BaseFileName {
+ private final String fileId;
+ private final String writeToken;
+ private final String commitTime;
+ private final String fileExtension;
+
+ private BaseFileName(String fileId, String writeToken, String commitTime, String fileExtension) {
+ this.fileId = fileId;
+ this.writeToken = writeToken;
+ this.commitTime = commitTime;
+ this.fileExtension = fileExtension;
+ }
+
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated log file name.
+ */
+ @Getter
+ public static final class LogFileName {
+ private final String fileId;
+ private final String deltaCommitTime;
+ private final int logVersion;
+ private final String writeToken;
+ private final Integer taskPartitionId;
+ private final Integer stageId;
+ private final Integer taskAttemptId;
+ private final String fileExtension;
+ // Inline logs use this as the optional CDC suffix (".cdc"), while native logs use it as
+ // the native storage format suffix, such as "parquet".
+ private final String suffix;
Review Comment:
🤖 nit: `suffix` carries unrelated data depending on log type — `.cdc` for inline CDC logs vs. the native storage format name (e.g. `parquet`) for native logs. Could you consider separate named fields or accessors (e.g. `cdcMarker` / `nativeFormat`) so callers don't need to guard with `isNativeLogFile()` before interpreting `getSuffix()`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.ExternalFilePathUtil;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+ private static final char UNDERSCORE = '_';
+ private static final char DOT = '.';
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>[.<extension>]}.
+ * Decoding mirrors {@code HoodieBaseFile}'s historical substring scan: the first underscore terminates
+ * the file id, the second underscore precedes the commit time, and the first dot after the second
+ * underscore terminates the commit time. If the dot is absent, the commit time extends to the end of
+ * the file name. The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot when
+ * an extension is present.</p>
+ *
+ * <p>Externally registered base files marked with {@code _hudiext} are decoded through
+ * {@link ExternalFilePathUtil}. For these files, {@link BaseFileName#getWriteToken()} is empty and
+ * {@link BaseFileName#getFileExtension()} is derived from the original external file name.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches a supported base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ if (ExternalFilePathUtil.isExternallyCreatedFile(actualFileName)) {
+ return parseExternalBaseFile(actualFileName);
+ }
+
+ int firstUnderscoreIndex = -1;
+ int secondUnderscoreIndex = -1;
+ int dotIndex = -1;
+ short underscoreCount = 0;
+ for (int i = 0; i < actualFileName.length(); i++) {
+ char c = actualFileName.charAt(i);
+ if (c == UNDERSCORE) {
+ if (underscoreCount == 0) {
+ firstUnderscoreIndex = i;
+ } else if (underscoreCount == 1) {
+ secondUnderscoreIndex = i;
+ }
+ underscoreCount++;
+ } else if (c == DOT && underscoreCount == 2) {
+ dotIndex = i;
+ break;
+ }
+ }
+ if (firstUnderscoreIndex < 0 || secondUnderscoreIndex < 0) {
+ return Option.empty();
+ }
+ int commitTimeEndIndex = dotIndex < 0 ? actualFileName.length() : dotIndex;
+ return Option.of(new BaseFileName(
+ actualFileName.substring(0, firstUnderscoreIndex),
+ actualFileName.substring(firstUnderscoreIndex + 1, secondUnderscoreIndex),
+ actualFileName.substring(secondUnderscoreIndex + 1, commitTimeEndIndex),
+ dotIndex < 0 ? "" : actualFileName.substring(dotIndex)));
+ }
+
+ /**
+ * Parses either an inline log file name or a native log file name.
+ *
+ * <p>Inline log format: {@code .<fileId>_<deltaCommitTime>.<extension>.<version>_<writeToken>[.cdc]}.
+ * Native log format: {@code <fileId>_<writeToken>_<deltaCommitTime>_<version>.<extension>.<nativeFormat>}.
+ * The decoded {@link LogFileName#getFileExtension()} value does not include a leading dot. For inline CDC
+ * logs, {@link LogFileName#getSuffix()} is {@code .cdc}; for native logs, it is the native storage format,
+ * such as {@code parquet}.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded log file name when the input matches either supported log-file format
+ */
+ public static Option<LogFileName> parseLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Option<LogFileName> nativeLogFileName = parseNativeLogFileFromActualFileName(actualFileName);
+ if (nativeLogFileName.isPresent()) {
+ return nativeLogFileName;
+ }
+
+ Matcher matcher = INLINE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(LogFileName.fromInlineLogFile(matcher)) : Option.empty();
+ }
+
+ /**
+ * Parses only a native log file name.
+ *
+ * <p>Use {@link #parseLogFile(String)} when both inline and native log formats are acceptable.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded native log file name when the input matches the native log-file format
+ */
+ public static Option<LogFileName> parseNativeLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ return parseNativeLogFileFromActualFileName(actualFileName);
+ }
+
+ /**
+ * Returns the file group prefix encoded in a Hudi file id.
+ *
+ * @param fileId Hudi file id ending with a numeric suffix separated by {@code -}
+ * @return file id prefix before the trailing numeric suffix
+ * @throws HoodieValidationException when the file id does not contain the expected suffix
+ */
+ public static String getFileIdPfxFromFileId(String fileId) {
+ Matcher matcher = PREFIX_BY_FILE_ID_PATTERN.matcher(fileId);
+ if (!matcher.find()) {
+ throw new HoodieValidationException("Failed to get prefix from " + fileId);
+ }
+ return matcher.group(1);
+ }
+
+ private static Option<LogFileName> parseNativeLogFileFromActualFileName(String actualFileName) {
+ return matchNativeLogFileFromActualFileName(actualFileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ private static Option<Matcher> matchNativeLogFileFromActualFileName(String actualFileName) {
+ Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(matcher) : Option.empty();
+ }
+
+ private static String getActualFileName(String fileName) {
+ return fileName.contains(StoragePath.SEPARATOR)
+ ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
+ : fileName;
+ }
+
+ private static Option<BaseFileName> parseExternalBaseFile(String actualFileName) {
+ String[] fileIdAndCommitTime = ExternalFilePathUtil.parseFileIdAndCommitTimeFromExternalFile(actualFileName);
+ return Option.of(new BaseFileName(fileIdAndCommitTime[0], "", fileIdAndCommitTime[1],
+ getFileExtension(fileIdAndCommitTime[0])));
+ }
+
+ private static String getFileExtension(String fileName) {
+ int lastSeparatorIndex = fileName.lastIndexOf(StoragePath.SEPARATOR);
+ int dotIndex = fileName.lastIndexOf(DOT);
+ return dotIndex > lastSeparatorIndex ? fileName.substring(dotIndex) : "";
+ }
+
+ /**
+ * Decoded parts of a base file name.
+ */
+ @Getter
+ public static final class BaseFileName {
+ private final String fileId;
+ private final String writeToken;
+ private final String commitTime;
+ private final String fileExtension;
+
+ private BaseFileName(String fileId, String writeToken, String commitTime, String fileExtension) {
+ this.fileId = fileId;
+ this.writeToken = writeToken;
+ this.commitTime = commitTime;
+ this.fileExtension = fileExtension;
+ }
+
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated log file name.
+ */
+ @Getter
+ public static final class LogFileName {
+ private final String fileId;
+ private final String deltaCommitTime;
+ private final int logVersion;
+ private final String writeToken;
+ private final Integer taskPartitionId;
+ private final Integer stageId;
+ private final Integer taskAttemptId;
+ private final String fileExtension;
+ // Inline logs use this as the optional CDC suffix (".cdc"), while native logs use it as
+ // the native storage format suffix, such as "parquet".
+ private final String suffix;
+ private final boolean nativeLogFile;
+
+ private LogFileName(String fileId, String deltaCommitTime, int logVersion, String writeToken,
+ Integer taskPartitionId, Integer stageId, Integer taskAttemptId,
+ String fileExtension, String suffix, boolean nativeLogFile) {
+ this.fileId = fileId;
+ this.deltaCommitTime = deltaCommitTime;
+ this.logVersion = logVersion;
+ this.writeToken = writeToken;
+ this.taskPartitionId = taskPartitionId;
+ this.stageId = stageId;
+ this.taskAttemptId = taskAttemptId;
+ this.fileExtension = fileExtension;
+ this.suffix = suffix;
+ this.nativeLogFile = nativeLogFile;
+ }
+
+ private static LogFileName fromInlineLogFile(Matcher matcher) {
+ String taskPartitionId = matcher.group(7);
+ String stageId = matcher.group(8);
+ String taskAttemptId = matcher.group(9);
+ return new LogFileName(
+ matcher.group(1),
+ matcher.group(2),
+ Integer.parseInt(matcher.group(4)),
+ matcher.group(6),
+ taskPartitionId == null ? null : Integer.parseInt(taskPartitionId),
+ stageId == null ? null : Integer.parseInt(stageId),
+ taskAttemptId == null ? null : Integer.parseInt(taskAttemptId),
+ matcher.group(3),
+ matcher.group(10) == null ? "" : matcher.group(10),
+ false);
+ }
+
+ private static LogFileName fromNativeLogFile(Matcher matcher) {
+ return new LogFileName(
+ matcher.group(1),
+ matcher.group(6),
+ Integer.parseInt(matcher.group(7)),
+ matcher.group(2),
+ Integer.parseInt(matcher.group(3)),
+ Integer.parseInt(matcher.group(4)),
+ Integer.parseInt(matcher.group(5)),
+ matcher.group(8),
+ matcher.group(9),
+ true);
+ }
+
+ public boolean isCDCLogFile() {
+ return nativeLogFile
+ ? HoodieCDCUtils.CDC_LOGFILE_SUFFIX.substring(1).equals(fileExtension)
Review Comment:
🤖 nit: the `.substring(1)` on `CDC_LOGFILE_SUFFIX` isn't obvious — it's stripping the leading dot because native log extensions are stored without one. A brief inline comment like `// native extensions omit the leading dot` would save a reader from having to look up what the constant looks like.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4910848459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
}, {
"hash" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"triggerType" : "PUSH"
}, {
"hash" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352)
* 4fca0f381d302adecfd56e4bf76937cc84e06117 UNKNOWN
* 31be9fcdb2d3389d720738b9202c6d34421ba619 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3536691577
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java:
##########
@@ -489,27 +425,21 @@ public static String makeNativeLogFileName(String fileId, String writeToken, Str
}
public static boolean isBaseFile(String path) {
- if (matchNativeLogFile(path).isPresent()) {
- return false;
- }
- String extension = getFileExtension(path);
- if (HoodieFileFormat.BASE_FILE_EXTENSIONS.contains(extension)) {
- return BASE_FILE_PATTERN.matcher(path).matches();
- }
- return false;
+ return FileNameParser.parseBaseFile(path)
+ .map(baseFileName -> HoodieFileFormat.BASE_FILE_EXTENSIONS.contains(baseFileName.getFileExtension()))
Review Comment:
Updated in 123d5692d10a. I kept this aligned with the original loose FSUtils behavior: parse the base-file name through FileNameParser, reject native log files, then check the decoded extension against HoodieFileFormat.BASE_FILE_EXTENSIONS. The one-underscore branch is preserved in FileNameParser so existing external/non-generated names like file_1.parquet continue to decode, and I added docs clarifying that this fallback exists for historical FSUtils compatibility and is not a Hudi-writer naming pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3528356248
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi-generated base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ static final Pattern BASE_FILE_PATTERN = Pattern.compile("[a-zA-Z0-9-]+_[a-zA-Z0-9-]+_[0-9]+\\.[a-zA-Z0-9]+");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a Hudi-generated base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>.<extension>}.
+ * The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot, matching existing
+ * base-file helper API behavior.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches the Hudi base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Matcher matcher = BASE_FILE_PATTERN.matcher(actualFileName);
+ if (!matcher.matches()) {
+ return Option.empty();
+ }
+
+ int firstUnderscoreIndex = actualFileName.indexOf('_');
Review Comment:
🤖 nit: `BASE_FILE_PATTERN` validates the format here but then the code re-parses the same string manually with `indexOf`. Could you add capture groups to the regex (e.g. `([a-zA-Z0-9-]+)_([a-zA-Z0-9-]+)_([0-9]+)(\.[a-zA-Z0-9]+)`) and use `matcher.group(N)` directly? That way the validator and extractor stay in sync automatically if the format ever changes.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi-generated base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ static final Pattern BASE_FILE_PATTERN = Pattern.compile("[a-zA-Z0-9-]+_[a-zA-Z0-9-]+_[0-9]+\\.[a-zA-Z0-9]+");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a Hudi-generated base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>.<extension>}.
+ * The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot, matching existing
+ * base-file helper API behavior.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches the Hudi base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Matcher matcher = BASE_FILE_PATTERN.matcher(actualFileName);
+ if (!matcher.matches()) {
+ return Option.empty();
+ }
+
+ int firstUnderscoreIndex = actualFileName.indexOf('_');
+ int secondUnderscoreIndex = actualFileName.indexOf('_', firstUnderscoreIndex + 1);
+ int dotIndex = actualFileName.indexOf('.', secondUnderscoreIndex + 1);
+ return Option.of(new BaseFileName(
+ actualFileName.substring(0, firstUnderscoreIndex),
+ actualFileName.substring(firstUnderscoreIndex + 1, secondUnderscoreIndex),
+ actualFileName.substring(secondUnderscoreIndex + 1, dotIndex),
+ actualFileName.substring(dotIndex)));
+ }
+
+ /**
+ * Parses either an inline log file name or a native log file name.
+ *
+ * <p>Inline log format: {@code .<fileId>_<deltaCommitTime>.<extension>.<version>_<writeToken>[.cdc]}.
+ * Native log format: {@code <fileId>_<writeToken>_<deltaCommitTime>_<version>.<extension>.<nativeFormat>}.
+ * The decoded {@link LogFileName#getFileExtension()} value does not include a leading dot. For inline CDC
+ * logs, {@link LogFileName#getSuffix()} is {@code .cdc}; for native logs, it is the native storage format,
+ * such as {@code parquet}.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded log file name when the input matches either supported log-file format
+ */
+ public static Option<LogFileName> parseLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Option<LogFileName> nativeLogFileName = parseNativeLogFileFromActualFileName(actualFileName);
+ if (nativeLogFileName.isPresent()) {
+ return nativeLogFileName;
+ }
+
+ Matcher matcher = INLINE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(LogFileName.fromInlineLogFile(matcher)) : Option.empty();
+ }
+
+ /**
+ * Parses only a native log file name.
+ *
+ * <p>Use {@link #parseLogFile(String)} when both inline and native log formats are acceptable.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded native log file name when the input matches the native log-file format
+ */
+ public static Option<LogFileName> parseNativeLogFile(String fileName) {
+ return matchNativeLogFile(fileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ /**
+ * Returns the file group prefix encoded in a Hudi file id.
+ *
+ * @param fileId Hudi file id ending with a numeric suffix separated by {@code -}
+ * @return file id prefix before the trailing numeric suffix
+ * @throws HoodieValidationException when the file id does not contain the expected suffix
+ */
+ public static String getFileIdPfxFromFileId(String fileId) {
+ Matcher matcher = PREFIX_BY_FILE_ID_PATTERN.matcher(fileId);
+ if (!matcher.find()) {
+ throw new HoodieValidationException("Failed to get prefix from " + fileId);
+ }
+ return matcher.group(1);
+ }
+
+ static Option<Matcher> matchNativeLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ return matchNativeLogFileFromActualFileName(actualFileName);
+ }
+
+ private static Option<LogFileName> parseNativeLogFileFromActualFileName(String actualFileName) {
+ return matchNativeLogFileFromActualFileName(actualFileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ private static Option<Matcher> matchNativeLogFileFromActualFileName(String actualFileName) {
+ Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(matcher) : Option.empty();
+ }
+
+ private static String getActualFileName(String fileName) {
+ return fileName.contains(StoragePath.SEPARATOR)
+ ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
+ : fileName;
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated base file name.
+ */
+ @Getter
+ public static final class BaseFileName {
+ private final String fileId;
+ private final String writeToken;
+ private final String commitTime;
+ private final String fileExtension;
+
+ private BaseFileName(String fileId, String writeToken, String commitTime, String fileExtension) {
+ this.fileId = fileId;
+ this.writeToken = writeToken;
+ this.commitTime = commitTime;
+ this.fileExtension = fileExtension;
+ }
+
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated log file name.
+ */
+ @Getter
+ public static final class LogFileName {
+ private final String fileId;
+ private final String deltaCommitTime;
+ private final int logVersion;
+ private final String writeToken;
+ private final Integer taskPartitionId;
+ private final Integer stageId;
+ private final Integer taskAttemptId;
+ private final String fileExtension;
+ private final String suffix;
Review Comment:
🤖 nit: the `suffix` field means different things depending on the log file type — it's `.cdc` (with a leading dot) for inline CDC log files, but `parquet` (no dot) for native log files, and `isCDCLogFile()` has to branch on `nativeLogFile` to account for this. Have you considered a rename like `cdcOrNativeFormat` with an inline comment, or splitting into two optional fields, to make this asymmetry explicit at the declaration site?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4892071987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8172034c4fe29ac4351e3fce3dda6e7af951f1d4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4891989479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8172034c4fe29ac4351e3fce3dda6e7af951f1d4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3529460837
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi-generated base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ static final Pattern BASE_FILE_PATTERN = Pattern.compile("[a-zA-Z0-9-]+_[a-zA-Z0-9-]+_[0-9]+\\.[a-zA-Z0-9]+");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a Hudi-generated base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>.<extension>}.
+ * The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot, matching existing
+ * base-file helper API behavior.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches the Hudi base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Matcher matcher = BASE_FILE_PATTERN.matcher(actualFileName);
+ if (!matcher.matches()) {
+ return Option.empty();
+ }
+
+ int firstUnderscoreIndex = actualFileName.indexOf('_');
Review Comment:
Thanks. We kept base-file parsing aligned with the historical HoodieBaseFile/origin-master substring scan instead of switching to regex group extraction, so this validator/extractor concern is no longer applicable in the current revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899736143
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3536188389
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java:
##########
@@ -489,27 +425,21 @@ public static String makeNativeLogFileName(String fileId, String writeToken, Str
}
public static boolean isBaseFile(String path) {
- if (matchNativeLogFile(path).isPresent()) {
- return false;
- }
- String extension = getFileExtension(path);
- if (HoodieFileFormat.BASE_FILE_EXTENSIONS.contains(extension)) {
- return BASE_FILE_PATTERN.matcher(path).matches();
- }
- return false;
+ return FileNameParser.parseBaseFile(path)
+ .map(baseFileName -> HoodieFileFormat.BASE_FILE_EXTENSIONS.contains(baseFileName.getFileExtension()))
Review Comment:
**`FSUtils.isBaseFile` is now more permissive than master**, which required a full `<fileId>_<writeToken>_<numericCommit>.<ext>` match. The lenient scan now accepts e.g. `foo_20240101.parquet` (single underscore, the "hacky" legacy branch) and `a_b_xyz.parquet` (non-numeric commit time).
This matters for listing-based classification: `HoodieRepairTool` treats `isDataFile` hits absent from commit metadata as dangling and deletes them in repair mode, and the MDT bootstrap listing (`HoodieTableMetadataUtil.DirectoryInfo`) would index a stray `foo_20240101.parquet` into the files partition as a phantom file group. Previously the strict pattern ignored such files.
Could we keep the stricter validation for non-external names here, and confine the lenient legacy branch to the `HoodieBaseFile` decoding path where the historical tolerance is actually needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4900731553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* dbded98f436830fc54b08dfbe4b4a064f03f5463 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318)
* aa3f77388cd859d4828e68fc2f5cc81acbe46fa6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4911390520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
}, {
"hash" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"triggerType" : "PUSH"
}, {
"hash" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"triggerType" : "PUSH"
}, {
"hash" : "bd2283fac7888c22734c75858caf83c3fd53e6d0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15377",
"triggerID" : "bd2283fac7888c22734c75858caf83c3fd53e6d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* 4fca0f381d302adecfd56e4bf76937cc84e06117 UNKNOWN
* 31be9fcdb2d3389d720738b9202c6d34421ba619 UNKNOWN
* bd2283fac7888c22734c75858caf83c3fd53e6d0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19198:
URL: https://github.com/apache/hudi/pull/19198#discussion_r3531561643
##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FileNameParser.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.fs;
+
+import org.apache.hudi.common.table.cdc.HoodieCDCUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.StoragePath;
+
+import lombok.Getter;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser for Hudi-generated base and log file names.
+ *
+ * <p>The parser accepts either a file name or a full path and always decodes only the last path component.
+ * A non-matching input returns {@link Option#empty()} instead of throwing.</p>
+ */
+public final class FileNameParser {
+ // Inline log files are of this pattern - .b5068208-e1a4-11e6-bf01-fe55135034f3_20170101134598.log.1_1-0-1
+ // Inline archive log files are of this pattern - .commits_.archive.1_1-0-1
+ // Native log files are of this pattern - b5068208-e1a4-11e6-bf01-fe55135034f3_1-0-1_20170101134598_1.log.parquet
+ // For native log files, the file extension is log/deletes/cdc and the suffix is the native file format.
+
+ static final Pattern INLINE_LOG_FILE_PATTERN =
+ Pattern.compile("^\\.([^._]+)_([^.]*)\\.(log|archive)\\.(\\d+)(_((\\d+)-(\\d+)-(\\d+))(\\.cdc)?)?$");
+ static final Pattern NATIVE_LOG_FILE_PATTERN =
+ Pattern.compile("^([^._]+)_((\\d+)-(\\d+)-(\\d+))_([^_]+)_(\\d+)\\.(log|deletes|cdc)\\.([^.]+)$");
+ private static final Pattern PREFIX_BY_FILE_ID_PATTERN = Pattern.compile("^(.+)-(\\d+)");
+ private static final char UNDERSCORE = '_';
+ private static final char DOT = '.';
+
+ private FileNameParser() {
+ }
+
+ /**
+ * Parses a Hudi-generated base file name.
+ *
+ * <p>Expected format: {@code <fileId>_<writeToken>_<commitTime>[.<extension>]}.
+ * Decoding mirrors {@code HoodieBaseFile}'s historical substring scan: the first underscore terminates
+ * the file id, the second underscore precedes the commit time, and the first dot after the second
+ * underscore terminates the commit time. If the dot is absent, the commit time extends to the end of
+ * the file name. The decoded {@link BaseFileName#getFileExtension()} value includes the leading dot when
+ * an extension is present.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded base file name when the input matches the Hudi base-file format
+ */
+ public static Option<BaseFileName> parseBaseFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ int firstUnderscoreIndex = -1;
+ int secondUnderscoreIndex = -1;
+ int dotIndex = -1;
+ short underscoreCount = 0;
+ for (int i = 0; i < actualFileName.length(); i++) {
+ char c = actualFileName.charAt(i);
+ if (c == UNDERSCORE) {
+ if (underscoreCount == 0) {
+ firstUnderscoreIndex = i;
+ } else if (underscoreCount == 1) {
+ secondUnderscoreIndex = i;
+ }
+ underscoreCount++;
+ } else if (c == DOT && underscoreCount == 2) {
+ dotIndex = i;
+ break;
+ }
+ }
+ if (firstUnderscoreIndex < 0 || secondUnderscoreIndex < 0) {
+ return Option.empty();
+ }
+ int commitTimeEndIndex = dotIndex < 0 ? actualFileName.length() : dotIndex;
+ return Option.of(new BaseFileName(
+ actualFileName.substring(0, firstUnderscoreIndex),
+ actualFileName.substring(firstUnderscoreIndex + 1, secondUnderscoreIndex),
+ actualFileName.substring(secondUnderscoreIndex + 1, commitTimeEndIndex),
+ dotIndex < 0 ? "" : actualFileName.substring(dotIndex)));
+ }
+
+ /**
+ * Parses either an inline log file name or a native log file name.
+ *
+ * <p>Inline log format: {@code .<fileId>_<deltaCommitTime>.<extension>.<version>_<writeToken>[.cdc]}.
+ * Native log format: {@code <fileId>_<writeToken>_<deltaCommitTime>_<version>.<extension>.<nativeFormat>}.
+ * The decoded {@link LogFileName#getFileExtension()} value does not include a leading dot. For inline CDC
+ * logs, {@link LogFileName#getSuffix()} is {@code .cdc}; for native logs, it is the native storage format,
+ * such as {@code parquet}.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded log file name when the input matches either supported log-file format
+ */
+ public static Option<LogFileName> parseLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ Option<LogFileName> nativeLogFileName = parseNativeLogFileFromActualFileName(actualFileName);
+ if (nativeLogFileName.isPresent()) {
+ return nativeLogFileName;
+ }
+
+ Matcher matcher = INLINE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(LogFileName.fromInlineLogFile(matcher)) : Option.empty();
+ }
+
+ /**
+ * Parses only a native log file name.
+ *
+ * <p>Use {@link #parseLogFile(String)} when both inline and native log formats are acceptable.</p>
+ *
+ * @param fileName file name or full path
+ * @return decoded native log file name when the input matches the native log-file format
+ */
+ public static Option<LogFileName> parseNativeLogFile(String fileName) {
+ if (StringUtils.isNullOrEmpty(fileName)) {
+ return Option.empty();
+ }
+
+ String actualFileName = getActualFileName(fileName);
+ return parseNativeLogFileFromActualFileName(actualFileName);
+ }
+
+ /**
+ * Returns the file group prefix encoded in a Hudi file id.
+ *
+ * @param fileId Hudi file id ending with a numeric suffix separated by {@code -}
+ * @return file id prefix before the trailing numeric suffix
+ * @throws HoodieValidationException when the file id does not contain the expected suffix
+ */
+ public static String getFileIdPfxFromFileId(String fileId) {
+ Matcher matcher = PREFIX_BY_FILE_ID_PATTERN.matcher(fileId);
+ if (!matcher.find()) {
+ throw new HoodieValidationException("Failed to get prefix from " + fileId);
+ }
+ return matcher.group(1);
+ }
+
+ private static Option<LogFileName> parseNativeLogFileFromActualFileName(String actualFileName) {
+ return matchNativeLogFileFromActualFileName(actualFileName).map(LogFileName::fromNativeLogFile);
+ }
+
+ private static Option<Matcher> matchNativeLogFileFromActualFileName(String actualFileName) {
+ Matcher matcher = NATIVE_LOG_FILE_PATTERN.matcher(actualFileName);
+ return matcher.matches() ? Option.of(matcher) : Option.empty();
+ }
+
+ private static String getActualFileName(String fileName) {
+ return fileName.contains(StoragePath.SEPARATOR)
+ ? fileName.substring(fileName.lastIndexOf(StoragePath.SEPARATOR) + 1)
+ : fileName;
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated base file name.
+ */
+ @Getter
+ public static final class BaseFileName {
+ private final String fileId;
+ private final String writeToken;
+ private final String commitTime;
+ private final String fileExtension;
+
+ private BaseFileName(String fileId, String writeToken, String commitTime, String fileExtension) {
+ this.fileId = fileId;
+ this.writeToken = writeToken;
+ this.commitTime = commitTime;
+ this.fileExtension = fileExtension;
+ }
+
+ }
+
+ /**
+ * Decoded parts of a Hudi-generated log file name.
+ */
+ @Getter
+ public static final class LogFileName {
+ private final String fileId;
+ private final String deltaCommitTime;
+ private final int logVersion;
+ private final String writeToken;
+ private final Integer taskPartitionId;
+ private final Integer stageId;
+ private final Integer taskAttemptId;
+ private final String fileExtension;
+ // Inline logs use this as the optional CDC suffix (".cdc"), while native logs use it as
Review Comment:
🤖 nit: `BaseFileName.getFileExtension()` returns the extension *with* the leading dot while `LogFileName.getFileExtension()` returns it *without* — that asymmetry is easy to trip over even though it's documented. Could you make the accessor names or return values consistent (or at least add a short reminder comment on each getter)?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4911004571
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
}, {
"hash" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15318",
"triggerID" : "dbded98f436830fc54b08dfbe4b4a064f03f5463",
"triggerType" : "PUSH"
}, {
"hash" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15323",
"triggerID" : "aa3f77388cd859d4828e68fc2f5cc81acbe46fa6",
"triggerType" : "PUSH"
}, {
"hash" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352",
"triggerID" : "123d5692d10aca83cc4f570d8e99e6ae9a69bdc7",
"triggerType" : "PUSH"
}, {
"hash" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fca0f381d302adecfd56e4bf76937cc84e06117",
"triggerType" : "PUSH"
}, {
"hash" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31be9fcdb2d3389d720738b9202c6d34421ba619",
"triggerType" : "PUSH"
}, {
"hash" : "bd2283fac7888c22734c75858caf83c3fd53e6d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15377",
"triggerID" : "bd2283fac7888c22734c75858caf83c3fd53e6d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
* 123d5692d10aca83cc4f570d8e99e6ae9a69bdc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15352)
* 4fca0f381d302adecfd56e4bf76937cc84e06117 UNKNOWN
* 31be9fcdb2d3389d720738b9202c6d34421ba619 UNKNOWN
* bd2283fac7888c22734c75858caf83c3fd53e6d0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899547640
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
}, {
"hash" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ec19f6d351e966dc3deaf28541bb08918cb8cbf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
* 8ec19f6d351e966dc3deaf28541bb08918cb8cbf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899520082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
}, {
"hash" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307",
"triggerID" : "0fbede460d20d30e95a9a43f8ba5f60f802fc80f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
* 0fbede460d20d30e95a9a43f8ba5f60f802fc80f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4899403874
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
}, {
"hash" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295",
"triggerID" : "2a93785726a9139ccc44a74df7dcd8ea23034201",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
* 2a93785726a9139ccc44a74df7dcd8ea23034201 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15295)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19198:
URL: https://github.com/apache/hudi/pull/19198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add FileNameParser for file name decoding [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19198:
URL: https://github.com/apache/hudi/pull/19198#issuecomment-4894057805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15263",
"triggerID" : "8172034c4fe29ac4351e3fce3dda6e7af951f1d4",
"triggerType" : "PUSH"
}, {
"hash" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268",
"triggerID" : "e186f84ac1b8dbd335715860808b92d7810c6e9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e186f84ac1b8dbd335715860808b92d7810c6e9b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15268)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org