You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "geserdugarov (via GitHub)" <gi...@apache.org> on 2024/10/24 10:50:51 UTC
[I] [SUPPORT] Wrong hash for buckets [hudi]
geserdugarov opened a new issue, #12155:
URL: https://github.com/apache/hudi/issues/12155
Current extraction of column values from key record for bucket hash is working wrong:
https://github.com/apache/hudi/blob/a7512a206c5a1e8ce251cac7a302632a57d8c848/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java#L135-L139
And it was done by #7342 .
**Describe the problem you faced**
We shouldn't just split by `,` and then result by `:`. It's a wrong algorithm.
In this case, if we have keys from timestamp only:
```SQL
DROP TABLE IF EXISTS hudi_ts_only_one_bucket;
CREATE TABLE hudi_ts_only_one_bucket (
ts STRING,
desc STRING,
PRIMARY KEY (ts) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = 'hdfs://some_path/hudi_ts_only_one_bucket',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'upsert',
'index.type'='BUCKET',
'hoodie.bucket.index.hash.field'='ts',
'hoodie.bucket.index.num.buckets'='3',
'write.tasks'='3',
'read.tasks'='3'
);
INSERT INTO hudi_ts_only_one_bucket VALUES
('2024-10-24 10:11:11.234','aa'),
('2024-10-24 10:22:12.345','aa'),
('2024-10-24 10:33:13.456','aa'),
('2024-10-24 10:44:14.567','aa'),
('2024-10-24 10:55:15.678','aa');
```
then all records will be placed in one bucket always despite any settings:
```Bash
hdfs dfs -ls -h hdfs://some_path/hudi_ts_only_one_bucket
Found 3 items
.../hudi_ts_only_one_bucket/.hoodie
.../hudi_ts_only_one_bucket/.hoodie_partition_metadata
.../hudi_ts_only_one_bucket/00000001-4a48-44f3-a3fa-63c7a537e4a1_1-3-0_20241024183249279.parquet
```
Or exactly opposite could happen. You want to place some data to one bucket by some field (`f2`):
```SQL
DROP TABLE IF EXISTS hudi_tricky_one_bucket;
CREATE TABLE hudi_tricky_one_bucket (
f1 STRING,
f2 STRING,
desc STRING,
PRIMARY KEY (f1,f2) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = 'hdfs://some_path/hudi_tricky_one_bucket',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'upsert',
'index.type'='BUCKET',
'hoodie.bucket.index.hash.field'='',
'hoodie.bucket.index.num.buckets'='3',
'write.tasks'='3',
'read.tasks'='3'
);
INSERT INTO hudi_tricky_one_bucket VALUES
('101,010','forceToOneBucket','aa'),
('122,120','forceToOneBucket','aa'),
('123,410','forceToOneBucket','aa');
```
but have a lot of buckets in a result:
```Bash
hdfs dfs -ls -h hdfs://some_path/hudi_should_be_one_but_got_a_lot
Found 5 items
.../hudi_should_be_one_but_got_a_lot/.hoodie
.../hudi_should_be_one_but_got_a_lot/.hoodie_partition_metadata
.../hudi_should_be_one_but_got_a_lot/00000000-b2fc-4e53-babd-10ee343463be_0-3-0_20241024184049044.parquet
.../hudi_should_be_one_but_got_a_lot/00000001-937b-409b-a2d1-e09b73c332ea_1-3-0_20241024184049044.parquet
.../hudi_should_be_one_but_got_a_lot/00000002-925f-4724-aaba-d32d9f8b526e_2-3-0_20241024184049044.parquet
```
**Expected behavior**
Customers expect for the case with timestamps to see as much buckets, as set by `hoodie.bucket.index.num.buckets`. And in the second case, it's expected that if `f2` has similar data, and it's used for bucketing, then we should have this data in similar bucket.
I have a fix, it's presented in https://github.com/apache/hudi/pull/12120. But the problem is with migration from current wrong hash to new one.
**Environment Description**
* Hudi version : current master, used 9b3f85e1b4d037a6e10c81e4b8d5f3e8a4a01ef6
* Flink version : 1.17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong hash for buckets [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434956415
For opposite example:
```Text
Key:f1:101,010,f2:forceToOneBucket; new hash: [forceToOneBucket]; current hash: [010, forceToOneBucket]; new bucket = 0; current bucket = 1
Key:f1:122,120,f2:forceToOneBucket; new hash: [forceToOneBucket]; current hash: [120, forceToOneBucket]; new bucket = 0; current bucket = 0
Key:f1:123,410,f2:forceToOneBucket; new hash: [forceToOneBucket]; current hash: [410, forceToOneBucket]; new bucket = 0; current bucket = 2
```
we didn't filter everything after `,` from `f1` despite that we want to hash only by `f2`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2440420439
+1 to fix it and release with 1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong hash for buckets [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434961223
But if we change hash function, there could be duplications due to some keys written to one bucket using current hash, and after hash fix, buckets could be different:
```Text
Key:f1:abc,f2:bcd,123; new hash: [abc, bcd,123]; current hash: [abc, bcd, 123]; new bucket = 2; current bucket = 1
Key:f1:hij,f2:jkl,,x; new hash: [hij, jkl,,x]; current hash: [hij, jkl, , x]; new bucket = 0; current bucket = 2
Key:f1:123,f2:234,567; new hash: [123, 234,567]; current hash: [123, 234, 567]; new bucket = 0; current bucket = 1
Key:f1:234,f2:,,7,89; new hash: [234, ,,7,89]; current hash: [234, , , 7, 89]; new bucket = 1; current bucket = 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong hash for buckets [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434948710
The main problem, that current filtering is really bad. I wrote some code for quick illustration:
```Java
package org.apache.hudi.index.bucket;
import org.apache.hudi.common.model.HoodieKey;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class TestBucketIdentifier {
public static String DEFAULT_RECORD_KEY_PARTS_SEPARATOR = ",";
public static String DEFAULT_COLUMN_VALUE_SEPARATOR = ":";
public static String NULL_RECORDKEY_PLACEHOLDER = "__null__";
public static String EMPTY_RECORDKEY_PLACEHOLDER = "__empty__";
@Test
public void testGetHashKeys() {
// for current hash, all timestamps to only one bucket
// List<String> recordKeys = Arrays.asList("2024-10-24 10:11:11.234", "2024-10-24 10:22:12.345", "2024-10-24 10:33:13.456", "2024-10-24 10:44:14.567", "2024-10-24 10:55:15.678");
// String indexKeyFields = "ts";
// current hash, all to only one bucket
// List<String> recordKeys = Arrays.asList("f1:101,010,f2:forceToOneBucket", "f1:122,120,f2:forceToOneBucket", "f1:123,410,f2:forceToOneBucket");
// String indexKeyFields = "f2";
// to illustrate problem with duplication considering migration for current hash to new one
List<String> recordKeys = Arrays.asList("f1:abc,f2:bcd,123", "f1:hij,f2:jkl,,x", "f1:123,f2:234,567", "f1:234,f2:,,7,89");
String indexKeyFields = "f1,f2";
int numBuckets = 3;
for (String recordKey : recordKeys) {
List<String> keysNew = newGetHashKeys(recordKey, indexKeyFields);
List<String> keysOld = currentGetHashKeys(recordKey, indexKeyFields);
int bucketNew = getBucketId(keysNew, numBuckets);
int bucketOld = getBucketId(keysOld, numBuckets);
System.out.println("Key:" + recordKey + ";\tnew hash: " + keysNew + ";\tcurrent hash: " + keysOld +
";\tnew bucket = " + bucketNew + ";\tcurrent bucket = " + bucketOld);
}
System.out.println("Finished");
}
public static int getBucketId(List<String> hashKeyFields, int numBuckets) {
return (hashKeyFields.hashCode() & Integer.MAX_VALUE) % numBuckets;
}
public static List<String> newGetHashKeys(String recordKey, String indexKeyFields) {
return Arrays.asList(newExtractRecordKeysByFields(recordKey, Arrays.asList(indexKeyFields.split(","))));
}
public static List<String> currentGetHashKeys(String recordKey, String indexKeyFields) {
return !recordKey.contains(":") ?
Collections.singletonList(recordKey) : Arrays.asList(currentExtractRecordKeysByFields(recordKey, Arrays.asList(indexKeyFields.split(","))));
}
public static String[] newExtractRecordKeysByFields(String recordKey, List<String> fields) {
// if there is no ',' and ':', then it's a key value
if (!recordKey.contains(DEFAULT_RECORD_KEY_PARTS_SEPARATOR) || !recordKey.contains(DEFAULT_COLUMN_VALUE_SEPARATOR)) {
return new String[] {recordKey};
}
// complex key case
// Here we're reducing memory allocation for substrings and use index positions,
// because for bucket index this will be called for each record, which leads to GC overhead
int keyValueSep1;
int keyValueSep2;
int commaPosition;
String currentField;
String currentValue;
List<String> values = new ArrayList<>();
int processed = 0;
while (processed < recordKey.length()) {
// note that keyValueSeps and commaPosition are absolute
keyValueSep1 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, processed);
currentField = recordKey.substring(processed, keyValueSep1);
keyValueSep2 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, keyValueSep1 + 1);
if (fields.isEmpty() || (fields.size() == 1 && fields.get(0).isEmpty()) || fields.contains(currentField)) {
if (keyValueSep2 < 0) {
// there is no next key value pair
currentValue = recordKey.substring(keyValueSep1 + 1);
processed = recordKey.length();
} else {
// looking for ',' in reverse order to support multiple ',' in key values by looking for the latest ','
commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
// commaPosition could be -1 if didn't find ',', or we could find ',' from previous key-value pair ('col1:val1,...')
// also we could have the last value with ':', so need to check if keyValueSep2 > 0
while (commaPosition < keyValueSep1 && keyValueSep2 > 0) {
// If we have key value as a timestamp with ':',
// then we continue to skip ':' until before the next ':' there is a ',' character.
// For instance, 'col1:val1,col2:2014-10-22 13:50:42,col3:val3'
// ^ ^ ^ ^
// 1) keyValueSep1 skip skip keyValueSep2
// ^
// commaPosition
// 2) | currentValue |
// ^
// 3) processed
keyValueSep2 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, keyValueSep2 + 1);
commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
}
if (commaPosition > 0) {
currentValue = recordKey.substring(keyValueSep1 + 1, commaPosition);
processed = commaPosition + 1;
} else {
// it could be the last value with many ':', in this case we wouldn't find any ',' before
currentValue = recordKey.substring(keyValueSep1 + 1);
processed = recordKey.length();
}
}
// here could be any logic of conditional replacing of currentValue
if (currentValue.equals(NULL_RECORDKEY_PLACEHOLDER)) {
values.add(null);
} else if (currentValue.equals(EMPTY_RECORDKEY_PLACEHOLDER)) {
values.add("");
} else {
values.add(currentValue);
}
} else {
if (keyValueSep2 < 0) {
processed = recordKey.length();
} else {
commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
while (commaPosition < keyValueSep1) {
// described above
keyValueSep2 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, keyValueSep2 + 1);
commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
}
if (commaPosition < 0) {
// if something went wrong, and there is no ',', we should stop here, and pass the whole recordKey,
// otherwise processed = commaPosition + 1 would lead to infinite loop
processed = recordKey.length();
} else {
processed = commaPosition + 1;
}
}
}
}
return values.isEmpty() ? new String[] {recordKey} : values.toArray(new String[0]);
}
public static String[] currentExtractRecordKeysByFields(String recordKey, List<String> fields) {
String[] fieldKV = recordKey.split(DEFAULT_RECORD_KEY_PARTS_SEPARATOR);
return Arrays.stream(fieldKV).map(kv -> kv.split(DEFAULT_COLUMN_VALUE_SEPARATOR, 2))
.filter(kvArray -> kvArray.length == 1 || fields.isEmpty() || (fields.contains(kvArray[0])))
.map(kvArray -> {
if (kvArray.length == 1) {
return kvArray[0];
} else if (kvArray[1].equals(NULL_RECORDKEY_PLACEHOLDER)) {
return null;
} else if (kvArray[1].equals(EMPTY_RECORDKEY_PLACEHOLDER)) {
return "";
} else {
return kvArray[1];
}
}).toArray(String[]::new);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2437418537
> In a result, current filtering for bucket hash is broken, and two examples above show that even for usual jobs, we could get unexpected distribution of data. But fixing of this filtering could lead to duplication of some data written before and after fix (if we wrote part of data using current hash, and continued after upgrade to write data using fixed hash).
>
> From my point of view there are not much possibilities:
>
> * to break for some specific scenarios backward compatibility, possibly attach it to 1.0 release, which already has drastical changes, (fix is ready in open MR [[HUDI-8403] Fixed values extraction for bucketing and optimized `KeyGenUtils::extractRecordKeysByFields` #12120](https://github.com/apache/hudi/pull/12120))
>
> * leave filtering as it is, with wrong behavior (I will change MR with optimization correspondingly).
@danny0405 @yihua @codope , if you don't mind, could you please, write your opinion about this situation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong hash for buckets [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434953604
And results of running shows, that for timestamps like `2024-10-24 10:11:11.234` we get column name `2024-10-24 10` and filter it, which leads to hash of `null` for all records.
```Text
Key:2024-10-24 10:11:11.234; new hash: [2024-10-24 10:11:11.234]; current hash: []; new bucket = 0; current bucket = 1
Key:2024-10-24 10:22:12.345; new hash: [2024-10-24 10:22:12.345]; current hash: []; new bucket = 2; current bucket = 1
Key:2024-10-24 10:33:13.456; new hash: [2024-10-24 10:33:13.456]; current hash: []; new bucket = 1; current bucket = 1
Key:2024-10-24 10:44:14.567; new hash: [2024-10-24 10:44:14.567]; current hash: []; new bucket = 0; current bucket = 1
Key:2024-10-24 10:55:15.678; new hash: [2024-10-24 10:55:15.678]; current hash: []; new bucket = 2; current bucket = 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov closed issue #12155: [SUPPORT] Wrong filtering of values for bucket hashes
URL: https://github.com/apache/hudi/issues/12155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong hash for buckets [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2435110468
In a result, current filtering for hash is broken, and two examples above show that even for usual jobs, we could get really badly distributed data. But fixing of this filtering could lead to duplications in result data, if we wrote part of it using current hash, and continued after upgrade to write data using fixed hash.
Maybe the only possibility is to attach this change to 1.0 release, if some backward compatibility is already broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2439175723
I'm +1 to fix it as a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org