You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinothchandar (via GitHub)" <gi...@apache.org> on 2026/01/24 00:06:48 UTC
[PR] refactor: drop unused InMemoryFileSystem class and test [hudi]
vinothchandar opened a new pull request, #17997:
URL: https://github.com/apache/hudi/pull/17997
### Describe the issue this Pull Request addresses
InMemoryFileSystem is unused now, since we pass in the outputstream object for both parquet and hfile writers. In current shape, it has also unresolved issues to support executors with num_cores >1. Dropping to avoid any accidents.
### Summary and Changelog
Deleted files (2 files, 267 lines):
- hudi-hadoop-common/src/main/java/org/apache/hudi/hadoop/fs/inline/InMemoryFileSystem.java - Dead production code
- hudi-hadoop-common/src/test/java/org/apache/hudi/common/fs/inline/TestInMemoryFileSystem.java - Tests for dead code
Refactored files (2 files):
- TestInLineFileSystemWithHFileReader.java - Now uses ByteArrayOutputStream instead of InMemoryFileSystem to generate HFile bytes
- TestParquetInLining.java - Now writes Parquet to a temp file on disk, reads the bytes, then proceeds with the InLineFileSystem test
Modified file:
- FileSystemTestUtils.java - Removed InMemoryFileSystem import and getRandomOuterInMemPath() method
Key Changes in Test Files
1. TestInLineFileSystemWithHFileReader: The HFileWriter now writes to a ByteArrayOutputStream directly (line 100-106), then embeds those bytes in the outer file. This preserves the test of reading HFile content via InLineFileSystem.
2. TestParquetInLining: Since ParquetWriter requires a Path, it now writes to a temp file on disk first (line 90-98), reads the bytes from that file (line 101), then embeds them in the outer file. The temp file is cleaned up in teardown().
Verification
- No references to org.apache.hudi.hadoop.fs.inline.InMemoryFileSystem remain
- No references to inmemfs scheme remain
- Both InLineFileSystem test files are preserved and refactored
### Impact
None. Dead code.
### Risk Level
None
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: drop unused InMemoryFileSystem class and test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17997:
URL: https://github.com/apache/hudi/pull/17997#issuecomment-3793100734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1d100f6320d9eb67d71a69ddadec762ad72b56d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11321",
"triggerID" : "1d100f6320d9eb67d71a69ddadec762ad72b56d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1d100f6320d9eb67d71a69ddadec762ad72b56d2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: drop unused InMemoryFileSystem class and test [hudi]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17997:
URL: https://github.com/apache/hudi/pull/17997#discussion_r2723243224
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/fs/inline/TestParquetInLining.java:
##########
@@ -73,22 +72,33 @@ public void teardown() throws IOException {
FileSystemTestUtils.deleteFile(filePath);
}
}
+ if (tempParquetPath != null) {
+ File tempFile = new File(tempParquetPath.toString().substring(tempParquetPath.toString().indexOf(':') + 1));
Review Comment:
The path extraction using substring and indexOf(':') is fragile and could fail with more complex URI formats. Consider using tempParquetPath.toUri().getPath() instead, which properly handles URI path extraction. This pattern is used elsewhere in the codebase (e.g., HoodieTestTable.java:907).
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/common/fs/inline/TestParquetInLining.java:
##########
@@ -120,9 +130,10 @@ private long generateOuterFile(Path outerPath, byte[] inlineBytes) throws IOExce
return startOffset;
}
- private byte[] getBytesToInline(Path outerInMemFSPath) throws IOException {
- InMemoryFileSystem inMemoryFileSystem = (InMemoryFileSystem) outerInMemFSPath.getFileSystem(inMemoryConf);
- return inMemoryFileSystem.getFileAsBytes();
+ private byte[] getBytesFromTempFile(Path tempPath) throws IOException {
+ String pathStr = tempPath.toString();
+ String filePath = pathStr.substring(pathStr.indexOf(':') + 1);
Review Comment:
The path extraction using substring and indexOf(':') is fragile and could fail with more complex URI formats. Consider using tempPath.toUri().getPath() instead, which properly handles URI path extraction. This pattern is used elsewhere in the codebase (e.g., HoodieTestTable.java:907).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: drop unused InMemoryFileSystem class and test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17997:
URL: https://github.com/apache/hudi/pull/17997#issuecomment-3793089897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1d100f6320d9eb67d71a69ddadec762ad72b56d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1d100f6320d9eb67d71a69ddadec762ad72b56d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1d100f6320d9eb67d71a69ddadec762ad72b56d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: drop unused InMemoryFileSystem class and test [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #17997:
URL: https://github.com/apache/hudi/pull/17997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: drop unused InMemoryFileSystem class and test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17997:
URL: https://github.com/apache/hudi/pull/17997#issuecomment-3793346826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1d100f6320d9eb67d71a69ddadec762ad72b56d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11321",
"triggerID" : "1d100f6320d9eb67d71a69ddadec762ad72b56d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1d100f6320d9eb67d71a69ddadec762ad72b56d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org