You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "geserdugarov (via GitHub)" <gi...@apache.org> on 2024/10/24 10:50:51 UTC

[I] [SUPPORT] Wrong hash for buckets [hudi]

geserdugarov opened a new issue, #12155:
URL: https://github.com/apache/hudi/issues/12155

   Current extraction of column values from key record for bucket hash is working wrong:
   https://github.com/apache/hudi/blob/a7512a206c5a1e8ce251cac7a302632a57d8c848/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java#L135-L139
   And it was done by #7342 . 
   
   **Describe the problem you faced**
   
   We shouldn't just split by `,` and then result by `:`. It's a wrong algorithm.
   
   In this case, if we have keys from timestamp only:
   ```SQL
   DROP TABLE IF EXISTS hudi_ts_only_one_bucket;
   
   CREATE TABLE hudi_ts_only_one_bucket (
       ts STRING,
       desc STRING,
       PRIMARY KEY (ts) NOT ENFORCED
   ) 
   WITH (
       'connector' = 'hudi',
       'path' = 'hdfs://some_path/hudi_ts_only_one_bucket',
       'table.type' = 'COPY_ON_WRITE',
       'write.operation' = 'upsert',
       'index.type'='BUCKET',
       'hoodie.bucket.index.hash.field'='ts',
       'hoodie.bucket.index.num.buckets'='3',
       'write.tasks'='3',
       'read.tasks'='3'
   );
   
   INSERT INTO hudi_ts_only_one_bucket VALUES 
       ('2024-10-24 10:11:11.234','aa'),
       ('2024-10-24 10:22:12.345','aa'),
       ('2024-10-24 10:33:13.456','aa'),
       ('2024-10-24 10:44:14.567','aa'),
       ('2024-10-24 10:55:15.678','aa');
   ```
   
   then all records will be placed in one bucket always despite any settings:
   ```Bash
   hdfs dfs -ls -h hdfs://some_path/hudi_ts_only_one_bucket
   Found 3 items
   .../hudi_ts_only_one_bucket/.hoodie
   .../hudi_ts_only_one_bucket/.hoodie_partition_metadata
   .../hudi_ts_only_one_bucket/00000001-4a48-44f3-a3fa-63c7a537e4a1_1-3-0_20241024183249279.parquet
   ```
   
   Or exactly opposite could happen. You want to place some data to one bucket by some field (`f2`):
   ```SQL
   DROP TABLE IF EXISTS hudi_tricky_one_bucket;
   
   CREATE TABLE hudi_tricky_one_bucket (
       f1 STRING,
       f2 STRING,
       desc STRING,
       PRIMARY KEY (f1,f2) NOT ENFORCED
   ) 
   WITH (
       'connector' = 'hudi',
       'path' = 'hdfs://some_path/hudi_tricky_one_bucket',
       'table.type' = 'COPY_ON_WRITE',
       'write.operation' = 'upsert',
       'index.type'='BUCKET',
       'hoodie.bucket.index.hash.field'='',
       'hoodie.bucket.index.num.buckets'='3',
       'write.tasks'='3',
       'read.tasks'='3'
   );
   
   INSERT INTO hudi_tricky_one_bucket VALUES 
       ('101,010','forceToOneBucket','aa'),
       ('122,120','forceToOneBucket','aa'),
       ('123,410','forceToOneBucket','aa');
   ```
   
   but have a lot of buckets in a result:
   ```Bash
   hdfs dfs -ls -h hdfs://some_path/hudi_should_be_one_but_got_a_lot
   Found 5 items
   .../hudi_should_be_one_but_got_a_lot/.hoodie
   .../hudi_should_be_one_but_got_a_lot/.hoodie_partition_metadata
   .../hudi_should_be_one_but_got_a_lot/00000000-b2fc-4e53-babd-10ee343463be_0-3-0_20241024184049044.parquet
   .../hudi_should_be_one_but_got_a_lot/00000001-937b-409b-a2d1-e09b73c332ea_1-3-0_20241024184049044.parquet
   .../hudi_should_be_one_but_got_a_lot/00000002-925f-4724-aaba-d32d9f8b526e_2-3-0_20241024184049044.parquet
   ```
   
   **Expected behavior**
   
   Customers expect for the case with timestamps to see as much buckets, as set by `hoodie.bucket.index.num.buckets`. And in the second case, it's expected that if `f2` has similar data, and it's used for bucketing, then we should have this data in similar bucket.
   
   I have a fix, it's presented in https://github.com/apache/hudi/pull/12120. But the problem is with migration from current wrong hash to new one.
   
   **Environment Description**
   
   * Hudi version : current master, used 9b3f85e1b4d037a6e10c81e4b8d5f3e8a4a01ef6
   
   * Flink version : 1.17
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong hash for buckets [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434956415

   For opposite example:
   ```Text
   Key:f1:101,010,f2:forceToOneBucket;	new hash: [forceToOneBucket];	current hash: [010, forceToOneBucket];	new bucket = 0;	current bucket = 1
   Key:f1:122,120,f2:forceToOneBucket;	new hash: [forceToOneBucket];	current hash: [120, forceToOneBucket];	new bucket = 0;	current bucket = 0
   Key:f1:123,410,f2:forceToOneBucket;	new hash: [forceToOneBucket];	current hash: [410, forceToOneBucket];	new bucket = 0;	current bucket = 2
   ```
   we didn't filter everything after `,` from `f1` despite that we want to hash only by `f2`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]

Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2440420439

   +1 to fix it and release with 1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong hash for buckets [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434961223

   But if we change hash function, there could be duplications due to some keys written to one bucket using current hash, and after hash fix, buckets could be different:
   ```Text
   Key:f1:abc,f2:bcd,123;	new hash: [abc, bcd,123];	current hash: [abc, bcd, 123];	new bucket = 2;	current bucket = 1
   Key:f1:hij,f2:jkl,,x;	new hash: [hij, jkl,,x];	current hash: [hij, jkl, , x];	new bucket = 0;	current bucket = 2
   Key:f1:123,f2:234,567;	new hash: [123, 234,567];	current hash: [123, 234, 567];	new bucket = 0;	current bucket = 1
   Key:f1:234,f2:,,7,89;	new hash: [234, ,,7,89];	current hash: [234, , , 7, 89];	new bucket = 1;	current bucket = 1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong hash for buckets [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434948710

   The main problem, that current filtering is really bad. I wrote some code for quick illustration:
   
   ```Java
   package org.apache.hudi.index.bucket;
   
   import org.apache.hudi.common.model.HoodieKey;
   
   import org.junit.jupiter.api.Test;
   
   import java.util.ArrayList;
   import java.util.Arrays;
   import java.util.Collections;
   import java.util.List;
   
   public class TestBucketIdentifier {
   
     public static String DEFAULT_RECORD_KEY_PARTS_SEPARATOR = ",";
     public static String DEFAULT_COLUMN_VALUE_SEPARATOR = ":";
     public static String NULL_RECORDKEY_PLACEHOLDER = "__null__";
     public static String EMPTY_RECORDKEY_PLACEHOLDER = "__empty__";
   
     @Test
     public void testGetHashKeys() {
       // for current hash, all timestamps to only one bucket
   //    List<String> recordKeys = Arrays.asList("2024-10-24 10:11:11.234", "2024-10-24 10:22:12.345", "2024-10-24 10:33:13.456", "2024-10-24 10:44:14.567", "2024-10-24 10:55:15.678");
   //    String indexKeyFields = "ts";
       // current hash, all to only one bucket
   //    List<String> recordKeys = Arrays.asList("f1:101,010,f2:forceToOneBucket", "f1:122,120,f2:forceToOneBucket", "f1:123,410,f2:forceToOneBucket");
   //    String indexKeyFields = "f2";
       // to illustrate problem with duplication considering migration for current hash to new one
       List<String> recordKeys = Arrays.asList("f1:abc,f2:bcd,123", "f1:hij,f2:jkl,,x", "f1:123,f2:234,567", "f1:234,f2:,,7,89");
       String indexKeyFields = "f1,f2";
       int numBuckets = 3;
       for (String recordKey : recordKeys) {
         List<String> keysNew = newGetHashKeys(recordKey, indexKeyFields);
         List<String> keysOld = currentGetHashKeys(recordKey, indexKeyFields);
         int bucketNew = getBucketId(keysNew, numBuckets);
         int bucketOld = getBucketId(keysOld, numBuckets);
         System.out.println("Key:" + recordKey + ";\tnew hash: " + keysNew + ";\tcurrent hash: " + keysOld +
             ";\tnew bucket = " + bucketNew + ";\tcurrent bucket = " + bucketOld);
       }
       System.out.println("Finished");
     }
     public static int getBucketId(List<String> hashKeyFields, int numBuckets) {
       return (hashKeyFields.hashCode() & Integer.MAX_VALUE) % numBuckets;
     }
   
     public static List<String> newGetHashKeys(String recordKey, String indexKeyFields) {
       return Arrays.asList(newExtractRecordKeysByFields(recordKey, Arrays.asList(indexKeyFields.split(","))));
     }
   
     public static List<String> currentGetHashKeys(String recordKey, String indexKeyFields) {
       return !recordKey.contains(":") ?
           Collections.singletonList(recordKey) : Arrays.asList(currentExtractRecordKeysByFields(recordKey, Arrays.asList(indexKeyFields.split(","))));
     }
   
     public static String[] newExtractRecordKeysByFields(String recordKey, List<String> fields) {
   
       // if there is no ',' and ':', then it's a key value
       if (!recordKey.contains(DEFAULT_RECORD_KEY_PARTS_SEPARATOR) || !recordKey.contains(DEFAULT_COLUMN_VALUE_SEPARATOR)) {
         return new String[] {recordKey};
       }
       // complex key case
       // Here we're reducing memory allocation for substrings and use index positions,
       // because for bucket index this will be called for each record, which leads to GC overhead
       int keyValueSep1;
       int keyValueSep2;
       int commaPosition;
       String currentField;
       String currentValue;
       List<String> values = new ArrayList<>();
       int processed = 0;
       while (processed < recordKey.length()) {
         // note that keyValueSeps and commaPosition are absolute
         keyValueSep1 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, processed);
         currentField = recordKey.substring(processed, keyValueSep1);
         keyValueSep2 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, keyValueSep1 + 1);
         if (fields.isEmpty() || (fields.size() == 1 && fields.get(0).isEmpty()) || fields.contains(currentField)) {
           if (keyValueSep2 < 0) {
             // there is no next key value pair
             currentValue = recordKey.substring(keyValueSep1 + 1);
             processed = recordKey.length();
           } else {
             // looking for ',' in reverse order to support multiple ',' in key values by looking for the latest ','
             commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
             // commaPosition could be -1 if didn't find ',', or we could find ',' from previous key-value pair ('col1:val1,...')
             // also we could have the last value with ':', so need to check if keyValueSep2 > 0
             while (commaPosition < keyValueSep1 && keyValueSep2 > 0) {
               // If we have key value as a timestamp with ':',
               // then we continue to skip ':' until before the next ':' there is a ',' character.
               // For instance, 'col1:val1,col2:2014-10-22 13:50:42,col3:val3'
               //                              ^             ^  ^       ^
               //   1)              keyValueSep1          skip  skip    keyValueSep2
               //                                                  ^
               //                                                  commaPosition
               //   2)                         |   currentValue    |
               //                                                  ^
               //   3)                                             processed
               keyValueSep2 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, keyValueSep2 + 1);
               commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
             }
             if (commaPosition > 0) {
               currentValue = recordKey.substring(keyValueSep1 + 1, commaPosition);
               processed = commaPosition + 1;
             } else {
               // it could be the last value with many ':', in this case we wouldn't find any ',' before
               currentValue = recordKey.substring(keyValueSep1 + 1);
               processed = recordKey.length();
             }
           }
           // here could be any logic of conditional replacing of currentValue
           if (currentValue.equals(NULL_RECORDKEY_PLACEHOLDER)) {
             values.add(null);
           } else if (currentValue.equals(EMPTY_RECORDKEY_PLACEHOLDER)) {
             values.add("");
           } else {
             values.add(currentValue);
           }
         } else {
           if (keyValueSep2 < 0) {
             processed = recordKey.length();
           } else {
             commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
             while (commaPosition < keyValueSep1) {
               // described above
               keyValueSep2 = recordKey.indexOf(DEFAULT_COLUMN_VALUE_SEPARATOR, keyValueSep2 + 1);
               commaPosition = recordKey.lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2);
             }
             if (commaPosition < 0) {
               // if something went wrong, and there is no ',', we should stop here, and pass the whole recordKey,
               // otherwise processed = commaPosition + 1 would lead to infinite loop
               processed = recordKey.length();
             } else {
               processed = commaPosition + 1;
             }
           }
         }
       }
       return values.isEmpty() ? new String[] {recordKey} : values.toArray(new String[0]);
     }
   
     public static String[] currentExtractRecordKeysByFields(String recordKey, List<String> fields) {
       String[] fieldKV = recordKey.split(DEFAULT_RECORD_KEY_PARTS_SEPARATOR);
       return Arrays.stream(fieldKV).map(kv -> kv.split(DEFAULT_COLUMN_VALUE_SEPARATOR, 2))
           .filter(kvArray -> kvArray.length == 1 || fields.isEmpty() || (fields.contains(kvArray[0])))
           .map(kvArray -> {
             if (kvArray.length == 1) {
               return kvArray[0];
             } else if (kvArray[1].equals(NULL_RECORDKEY_PLACEHOLDER)) {
               return null;
             } else if (kvArray[1].equals(EMPTY_RECORDKEY_PLACEHOLDER)) {
               return "";
             } else {
               return kvArray[1];
             }
           }).toArray(String[]::new);
     }
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2437418537

   > In a result, current filtering for bucket hash is broken, and two examples above show that even for usual jobs, we could get unexpected distribution of data. But fixing of this filtering could lead to duplication of some data written before and after fix (if we wrote part of data using current hash, and continued after upgrade to write data using fixed hash).
   > 
   > From my point of view there are not much possibilities:
   > 
   >     * to break for some specific scenarios backward compatibility, possibly attach it to 1.0 release, which already has drastical changes, (fix is ready in open MR [[HUDI-8403] Fixed values extraction for bucketing and optimized `KeyGenUtils::extractRecordKeysByFields` #12120](https://github.com/apache/hudi/pull/12120))
   > 
   >     * leave filtering as it is, with wrong behavior (I will change MR with optimization correspondingly).
   
   @danny0405 @yihua @codope , if you don't mind, could you please, write your opinion about this situation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong hash for buckets [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2434953604

   And results of running shows, that for timestamps like `2024-10-24 10:11:11.234` we get column name `2024-10-24 10` and filter it, which leads to hash of `null` for all records.
   ```Text
   Key:2024-10-24 10:11:11.234;	new hash: [2024-10-24 10:11:11.234];	current hash: [];	new bucket = 0;	current bucket = 1
   Key:2024-10-24 10:22:12.345;	new hash: [2024-10-24 10:22:12.345];	current hash: [];	new bucket = 2;	current bucket = 1
   Key:2024-10-24 10:33:13.456;	new hash: [2024-10-24 10:33:13.456];	current hash: [];	new bucket = 1;	current bucket = 1
   Key:2024-10-24 10:44:14.567;	new hash: [2024-10-24 10:44:14.567];	current hash: [];	new bucket = 0;	current bucket = 1
   Key:2024-10-24 10:55:15.678;	new hash: [2024-10-24 10:55:15.678];	current hash: [];	new bucket = 2;	current bucket = 1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov closed issue #12155: [SUPPORT] Wrong filtering of values for bucket hashes
URL: https://github.com/apache/hudi/issues/12155


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong hash for buckets [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2435110468

   In a result, current filtering for hash is broken, and two examples above show that even for usual jobs, we could get really badly distributed data. But fixing of this filtering could lead to duplications in result data, if we wrote part of it using current hash, and continued after upgrade to write data using fixed hash.
   Maybe the only possibility is to attach this change to 1.0 release, if some backward compatibility is already broken.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Wrong filtering of values for bucket hashes [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12155:
URL: https://github.com/apache/hudi/issues/12155#issuecomment-2439175723

   I'm +1 to fix it as a bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org