You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jspaine (via GitHub)" <gi...@apache.org> on 2023/07/24 09:43:29 UTC

[GitHub] [hudi] jspaine opened a new issue, #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

jspaine opened a new issue, #9271:
URL: https://github.com/apache/hudi/issues/9271

   **Describe the problem you faced**
   
   Enabling the bloom index metadata options below on a large partitioned table causes upserted rows to be duplicated and deletions to fail to remove most records. Re-creating the table unpartitioned, or with the index metadata options disabled, resolves the issue.
   
   ```
   "hoodie.bloom.index.use.metadata": "true",
   "hoodie.metadata.index.bloom.filter.enable": "true",
   ```
   
   **To Reproduce**
   
   I'm not sure how easy it is to reproduce, but it was very repeatable with my dataset:
   
   The table is almost 500M rows, 104 columns, partitioned by year, month into ~150 partitions. There are a few hundred rows up to 2015 then exponentially increasing with half the total rows in the current year.
   
   It seemed that in each commit, most of the rows to be updated were inserted as duplicates, and most rows to be deleted were not deleted. Checking one of the (hundreds of thousands) of duplicated rows showed the records are in separate files, and the filename of the original record didn't appear in the commit log:
   
   ![20230720_09h00m15s_grim](https://github.com/apache/hudi/assets/6288863/f89983e3-733b-4a96-bca1-f679de4357d5)
   
   
   Just tried again with the dev version of the table (3M rows with much slower growth and less partition skewing), doing a handful of updates/deletes, and still can't see the issue there. 
   
   Steps to reproduce the behavior:
   
   1. Load existing data using insert with index metadata options enabled
   2. Apply a batch of updates
   3. Select rows grouped by record key having count > 1
   4. See that many of the records that should have been updated have been duplicated
   
   Insert options:
   ```python
   {
           "hoodie.bloom.index.use.metadata": "true",
           "hoodie.database.name": "hudi_db",
           "hoodie.datasource.hive_sync.database": "hudi_db",
           "hoodie.datasource.hive_sync.enable": "true",
           "hoodie.datasource.hive_sync.mode": "hms",
           "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
           "hoodie.datasource.hive_sync.partition_fields": "partition_year,partition_month",
           "hoodie.datasource.hive_sync.table_properties": "hudi.metadata-listing-enabled=TRUE",
           "hoodie.datasource.hive_sync.table": "table",
           "hoodie.datasource.meta_sync.condition.sync": "true",
           "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
           "hoodie.datasource.write.operation": "insert",
           "hoodie.datasource.write.partitionpath.field": "partition_year,partition_month",
           "hoodie.datasource.write.payload.class": "org.apache.hudi.common.model.DefaultHoodieRecordPayload",
           "hoodie.datasource.write.precombine.field": "_timestamp",
           "hoodie.datasource.write.reconcile.schema": "true",
           "hoodie.datasource.write.recordkey.field": "_id",
           "hoodie.enable.data.skipping": "true",
           "hoodie.index.type": "BLOOM",
           "hoodie.insert.shuffle.parallelism": "300",
           "hoodie.metadata.index.bloom.filter.enable": "true",
           "hoodie.metadata.index.column.stats.enable": "true",
           "hoodie.parquet.compression.codec": "SNAPPY",
           "hoodie.parquet.compression.ratio": "0.75",
           "hoodie.payload.ordering.field": "_timestamp",
           "hoodie.schema.on.read.enable": "true",
           "hoodie.table.name": "table",
           "path": f"s3://{config.output_bucket}/{TABLE_NAME}",
   }
   ```
   
   Upsert options:
   ```python
   {
   	"hoodie.bloom.index.use.metadata": "true",
           "hoodie.database.name": "hudi_db",
   	"hoodie.datasource.hive_sync.database": "hudi_db",
   	"hoodie.datasource.hive_sync.enable": "true",
   	"hoodie.datasource.hive_sync.mode": "hms",
   	"hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
   	"hoodie.datasource.hive_sync.partition_fields": "partition_year,partition_month",
   	"hoodie.datasource.hive_sync.table_properties": "hudi.metadata-listing-enabled=TRUE",
   	"hoodie.datasource.hive_sync.table": "table",
   	"hoodie.datasource.meta_sync.condition.sync": "true",
   	"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
   	"hoodie.datasource.write.partitionpath.field": "partition_year,partition_month",
   	"hoodie.datasource.write.payload.class": "org.apache.hudi.common.model.DefaultHoodieRecordPayload",
   	"hoodie.datasource.write.precombine.field": "_timestamp",
   	"hoodie.datasource.write.reconcile.schema": "true",
   	"hoodie.datasource.write.recordkey.field": "_id",
   	"hoodie.enable.data.skipping": "true",
   	"hoodie.index.type": "BLOOM",
   	"hoodie.metadata.index.bloom.filter.enable": "true",
   	"hoodie.metadata.index.column.stats.enable": "true",
   	"hoodie.metrics.cloudwatch.metric.prefix": "hudi_db.table",
   	"hoodie.metrics.on": "true" if config.enable_monitoring else "false",
   	"hoodie.metrics.reporter.type": "CLOUDWATCH",
   	"hoodie.parquet.compression.codec": "SNAPPY",
   	"hoodie.parquet.compression.ratio": "0.75",
   	"hoodie.payload.ordering.field": "_timestamp",
   	"hoodie.schema.on.read.enable": "true",
   	"hoodie.table.name": "table",
   	"hoodie.upsert.shuffle.parallelism": "100",
   	"path": f"s3://{config.output_bucket}/{TABLE_NAME}",
   }
   ```
   
   **Expected behavior**
   
   No duplicates are produced when updating records
   
   **Environment Description**
   
   * Hudi version : 0.13.0 (EMR serverless 6.11.0, also tried hudi 0.12.2 on EMR serverless 6.10.0)
   
   * Spark version : 3.3.2
   
   * Hive version : 3.1.3
   
   * Hadoop version : 3.3.3
   
   * Storage (HDFS/S3/GCS..) : S3
   
   * Running on Docker? (yes/no) : yes
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled
URL: https://github.com/apache/hudi/issues/9271


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1647758062

   @jspaine 
   When you were running insert, is your source containing duplicates?
   If yes, can you try using hoodie.datasource.write.operation as upsert only for the first case also. upsert deduplicates the records.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] jspaine commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "jspaine (via GitHub)" <gi...@apache.org>.
jspaine commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1649254653

   @ad1happy2go no the inserted df doesn't contain duplicates, trying again with `--conf spark.sql.codegen.maxFields=150`:
   ```sql
   select 
       count(*) total,
       count(distinct _id) ids
   from hudi_db.table
   
   -- after insert:
   -- #	total	        ids
   -- 1	500849433	500849433
   
   -- after upsert:
   -- #	total	        ids
   -- 1	500906271	500859829
   ```
   
   Inserts/updates in the upsert commit:
   ```sql
   select op, count(*)
   from hudi_db.table
   where _hoodie_commit_time = (select max(_hoodie_commit_time) from hudi_db.table)
   group by op
   
   -- #	op	_col1
   -- 1	I	14263
   -- 2	U	42742
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1647714129

   cc @ad1happy2go for the help ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1649912752

   @jspaine 
   We need to test out "hoodie.bloom.index.use.metadata" with scale.
   Meanwhile to mitigate this issue can you try turn it to false and check if you still see the duplicates. 
   I will work on reproducing this meanwhile by testing this out on scale.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1648153165

   @jspaine As number of columns in dataset is 104, Can you try setting spark.sql.codegen.maxFields to more than that. Default value is 100.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] jspaine commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "jspaine (via GitHub)" <gi...@apache.org>.
jspaine commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1667305763

   @ad1happy2go  yes, it didn't seem to make any difference


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1764814154

   Thanks @jspaine . We were able to reproduce this issue as part of https://github.com/apache/hudi/issues/9870
   We will be working on a fix here. 
   
   Tracking JIRA - https://issues.apache.org/jira/browse/HUDI-6946
   
   We will track as part of that issue and JIRA. So closing out this. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] jspaine commented on issue #9271: [SUPPORT] Duplicates upserting into large partitioned table with bloom index metadata enabled

Posted by "jspaine (via GitHub)" <gi...@apache.org>.
jspaine commented on issue #9271:
URL: https://github.com/apache/hudi/issues/9271#issuecomment-1653155208

   @ad1happy2go thanks, yeah it's fine with that config disabled.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org