You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 07:53:22 UTC
[I] Make maps in HoodieTableFileSystemView concurrent maps [hudi]
hudi-bot opened a new issue, #15971:
URL: https://github.com/apache/hudi/issues/15971
{code:java}
// mapping from partition paths to file groups contained within them
protected Map<String, List<HoodieFileGroup>> partitionToFileGroupsMap;
/**
* PartitionPath + File-Id to pending compaction instant time.
*/
protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>> fgIdToPendingCompaction;
/**
* PartitionPath + File-Id to pending compaction instant time.
*/
protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>> fgIdToPendingLogCompaction;
/**
* PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped).
*/
protected Map<HoodieFileGroupId, BootstrapBaseFileMapping> fgIdToBootstrapBaseFile;
/**
* Track replace time for replaced file groups.
*/
protected Map<HoodieFileGroupId, HoodieInstant> fgIdToReplaceInstants;
/**
* Track file groups in pending clustering.
*/
protected Map<HoodieFileGroupId, HoodieInstant> fgIdToPendingClustering; {code}
The maps should be ConcurrentMaps, because addFilesToView in AbstractTableFileSystem can be called on multiple partitions concurrently.
SpillableMapBasedFileSystemView uses ExternalSpillableMap which is not thread safe, so we cannot update HoodieTableFileSystemView because then SpillableMapBasedFileSystemView would not implement it correctly
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6249
- Type: Improvement
---
## Comments
24/May/23 18:09;jonvex;accidentally closed;;;
---
03/Apr/25 00:24;tim.brown;The issue here is that the write and read locks were not always being used when required. Raised a PR with locking updates to handle this https://github.com/apache/hudi/pull/12982;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make maps in HoodieTableFileSystemView concurrent maps [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15971:
URL: https://github.com/apache/hudi/issues/15971#issuecomment-3630125801
### Linked PR(s)
- https://github.com/apache/hudi/pull/13082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make maps in HoodieTableFileSystemView concurrent maps [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15971:
URL: https://github.com/apache/hudi/issues/15971#issuecomment-4795510493
Triage: this looks resolved on current master. The six maps named in the issue body (`partitionToFileGroupsMap`, `fgIdToPendingCompaction`, `fgIdToPendingLogCompaction`, `fgIdToBootstrapBaseFile`, `fgIdToReplaceInstants`, `fgIdToPendingClustering`) are all instantiated as `ConcurrentHashMap` in `HoodieTableFileSystemView.java` (lines ~160-185) via [HUDI-6201](https://issues.apache.org/jira/browse/HUDI-6201) / commit 6b24ab1 (PR #8782, May 2023).
One small follow-up: there's a stale `//TODO: [HUDI-6249]` comment still sitting at line 59 of `HoodieTableFileSystemView.java` — the code below it already does what it asks, so it can be removed.
The broader read/write-lock discipline concern raised in the reopen comment (PR #12982 / #13082) is a separate, larger discussion than the original title asks for; could that be tracked under its own JIRA so this one can be closed? cc @jonvex @the-other-tim-brown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Make maps in HoodieTableFileSystemView concurrent maps [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15971: Make maps in HoodieTableFileSystemView concurrent maps
URL: https://github.com/apache/hudi/issues/15971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org