You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/02/09 11:00:00 UTC
[jira] [Commented] (IMPALA-14623) Use the raw bytes of the 128-bit Murmur hash of Iceberg file paths
[ https://issues.apache.org/jira/browse/IMPALA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18057328#comment-18057328 ]
ASF subversion and git services commented on IMPALA-14623:
----------------------------------------------------------
Commit 49196804f8e74db346001d951b44777d6e0b2bfe in impala's branch refs/heads/master from Arnab Karmakar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=49196804f ]
IMPALA-14623: Optimize memory usage for Iceberg file path hashes
Use THash128 Thrift struct (16 bytes) instead of String (64 bytes)
for storing 128-bit XXH128 hashes of Iceberg file paths,
achieving 4x memory reduction.
Key Changes:
- Added THash128 Thrift struct with two i64 fields (high/low)
- Updated TIcebergContentFileStore to use THash128 as map keys
- Created Hash128 Java class with Thrift serialization support
- Migrated from Murmur3 to XXH128 for better performance
- Added C++ comparison operators for THash128
Testing:
- Added comprehensive JUnit tests for Hash128 class
Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1
Reviewed-on: http://gerrit.cloudera.org:8080/23946
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
> Use the raw bytes of the 128-bit Murmur hash of Iceberg file paths
> ------------------------------------------------------------------
>
> Key: IMPALA-14623
> URL: https://issues.apache.org/jira/browse/IMPALA-14623
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Arnab Karmakar
> Priority: Major
> Labels: impala-iceberg
>
> Currently we use the following method to store the hash of an Iceberg file path:
>
> In IcebergUtil:
> {noformat}
> public static String getFilePathHash(String path) {
> Hasher hasher = Hashing.murmur3_128().newHasher();
> hasher.putUnencodedChars(path);
> return hasher.hash().toString();
> }{noformat}
> There are 16 raw bytes, but the String representation stores it on 2 * 16 = 32 characters. And a character in java is 2 bytes. So it consumes 2 * 32 = 64 bytes which is 4 times more than needed.
> For tables with large number of files this can cause a significant overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org