You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "wqwl611 (via GitHub)" <gi...@apache.org> on 2024/03/19 06:51:16 UTC
[I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
wqwl611 opened a new issue, #10882:
URL: https://github.com/apache/hudi/issues/10882
I want to upgrade Hudi from 0.11.1 to 0.13.1, but I encountered the problem of duplicate data. I have never encountered it before with the same configuration.
<img width="1634" alt="image" src="https://github.com/apache/hudi/assets/67826098/9f10b00b-c47f-47dd-959c-1941998614d5">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "wqwl611 (via GitHub)" <gi...@apache.org>.
wqwl611 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2011369007
> @wqwl611 Do you also share the timeline. I see both the records you pasted in screenshot belongs to same file group and both the records are in same file. Couple of questions -
>
> 1. Any possibility that the job could have run multi writer jobs without lock configuration?
> 2. Did you used bulk insert to ingest to this table initially?
> 3. Do you see any executor failures or retries after that you started seeing dups.
>
> Is this issue occurring frequently now, or only after the upgrade you see this happened.
@ad1happy2go the timeline is archived......
1. no possibility， but I found that I inited each table in single-writer mode. is this all right?
2.no, I used only upsert.
3. I didn't see any failures, because this issue occurred rarely.
4. I encounterd this issue only after the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "wqwl611 (via GitHub)" <gi...@apache.org>.
wqwl611 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2006238293
> can you also share the basic write configs
@danny0405 updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2049458945
@wqwl611 Can you please keep us posted on this in case you find anything here. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2006118575
Can you share you basic infrustructure, like the release versions of each component etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "wqwl611 (via GitHub)" <gi...@apache.org>.
wqwl611 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2046845414
> @wqwl611 I tried the same configuration but unable to reproduce. As you also mentioned that you are also getting this in very few cases. So we need to verify this on your env. In case you are there in hudi slack can you ping me and we can connect on call. (Aditya Goenka)
@ad1happy2go thanks for your help，but the company's environment is not convenient for sharing, we are trying other types of indexes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "wqwl611 (via GitHub)" <gi...@apache.org>.
wqwl611 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2006163300
> Can you share you basic infrustructure, like the release versions of each component etc.
@danny0405 updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2043992885
hey @ad1happy2go : if this is related to MDT, can you let me know. I am trying to take stock of all MDT data consistency related issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "wqwl611 (via GitHub)" <gi...@apache.org>.
wqwl611 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2006766143
> @danny0405 Can you give more details on how did you ingested this table? What writer configuration you used and did you changed index type for this table?
@ad1happy2go
I write this table by spark like "df.write.format("org.apache.hudi").save(basePath)", and I use multi-writer mode and a independent compactor service。
Following is my full write config:
df.write.format("org.apache.hudi").
// base info
option(TABLE_TYPE.key(), tableType).
option(OPERATION.key(), opType).
option(TBL_NAME.key, tableNameWithSuffix).
option(HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key, upsertParallelism).
option(HoodieWriteConfig.INSERT_PARALLELISM_VALUE.key, upsertParallelism).
option(HoodieWriteConfig.BULKINSERT_PARALLELISM_VALUE.key, upsertParallelism).
option(PRECOMBINE_FIELD.key, preCombineField).
option(EVENT_TIME_FIELD.key, preCombineField).
option(ORDERING_FIELD.key, preCombineField).
option(PARTITIONPATH_FIELD.key, partitionFields).
option(RECORDKEY_FIELD.key, recordKeyField).
// key
option(HoodieWriteConfig.KEYGENERATOR_TYPE.key, KeyGeneratorType.COMPLEX.name).
option(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key, classOf[ComplexKeyGenerator].getName).
// hive info
option(HiveSyncConfig.HIVE_SYNC_ENABLED.key(), syncHive).
option(HiveSyncConfig.HIVE_SYNC_TABLE_STRATEGY.key(), "RT").
option(HIVE_STYLE_PARTITIONING.key(), true).
option(HiveSyncConfig.HIVE_IGNORE_EXCEPTIONS.key(), false).
option(HiveSyncConfig.HIVE_CREATE_MANAGED_TABLE.key(), false).
option(HiveSyncConfig.HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE.key(), "true").
option(HiveSyncConfig.HIVE_SYNC_MODE.key(), "hms").
option(HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS.key, classOf[MultiPartKeysValueExtractor].getName).
option(HoodieSyncConfig.META_SYNC_DATABASE_NAME.key(), hiveDb).
option(HoodieSyncConfig.META_SYNC_TABLE_NAME.key(), hiveTableName).
// metadata table 关闭 元数据表
options(metatableOptions).
// compact info
option(ASYNC_COMPACT_ENABLE.key, true).
option(INLINE_COMPACT.key, inLineCompact).
option(SCHEDULE_INLINE_COMPACT.key, scheduleInlineCompact).
option(COMPACTION_LAZY_BLOCK_READ_ENABLE.key, true).
option(COMPACTION_LOG_FILE_SIZE_THRESHOLD.key, CompactorParamUtils.getLogSizeThreshold(tableNameWithSuffix)).
option(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE.key, maxFileSize).
option(PARQUET_SMALL_FILE_LIMIT.key, minLimitFileSize).
option(HoodieStorageConfig.PARQUET_COMPRESSION_CODEC_NAME.key, "gzip").
// archive
option(HoodieArchivalConfig.MAX_COMMITS_TO_KEEP.key, "180").
option(HoodieArchivalConfig.MIN_COMMITS_TO_KEEP.key, "120").
option(HoodieArchivalConfig.COMMITS_ARCHIVAL_BATCH_SIZE.key, "100").
option(HoodieArchivalConfig.AUTO_ARCHIVE.key, "true").
option(HoodieArchivalConfig.ASYNC_ARCHIVE.key, "false").
// important payload.....
option(HoodiePayloadConfig.PAYLOAD_CLASS_NAME.key, classOf[EventTimeAvroPayload].getName).
option(HoodieWriteConfig.WRITE_PAYLOAD_CLASS_NAME.key, classOf[EventTimeAvroPayload].getName).
option(HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key, CompactionTriggerStrategy.NUM_COMMITS.name).
option(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS.key, CompactorParamUtils.getMaxDeltaCommits(tableNameWithSuffix)).
option(HoodieCompactionConfig.INLINE_COMPACT_TIME_DELTA_SECONDS.key, 3600 * 3).
// index info
options(buildCleanerOptions(df.sparkSession)).
option(HoodieIndexConfig.INDEX_TYPE.key, IndexType.BUCKET.name()).
option(HoodieIndexConfig.BUCKET_INDEX_NUM_BUCKETS.key, 1).
option(HoodieIndexConfig.BUCKET_INDEX_HASH_FIELD.key, hashKey).
option(HoodieLayoutConfig.LAYOUT_TYPE.key, "BUCKET").
option(HoodieLayoutConfig.LAYOUT_PARTITIONER_CLASS_NAME.key, classOf[SparkBucketIndexPartitioner[_]].getName).
// 并发
options(buildConcurrentOptions(tableNameWithSuffix)).
// heartbeat
option(HoodieWriteConfig.CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES.key, 10).
option(HoodieWriteConfig.CLIENT_HEARTBEAT_INTERVAL_IN_MS.key, 60000).
mode(Append).
save(basePath)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2007250457
If you are using multiwriter, which lock provider you are using? I cant see the lock configuration in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2006733479
@danny0405 Can you give more details on how did you ingested this table? What writer configuration you used and did you changed index type for this table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2015093418
@wqwl611 I tried the same configuration but unable to reproduce. As you also mentioned that you are also getting this in very few cases. So we need to verify this on your env. In case you are there in hudi slack can you ping me and we can connect on call. (Aditya Goenka)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-4852587890
Since this started only after the 0.11.1 → 0.13.1 upgrade with the same config that was clean before, and both dups are in the same file group/base file, this looks like an intermittent index-tagging miss — the upsert occasionally fails to tag an existing key as an update and inserts a duplicate, which compaction then merges into the base file. Given nsivabalan's note, the metadata table is the prime suspect: a jar-swap upgrade re-bootstraps MDT over existing data, and a stale MDT index (esp. bloom filters) can cause false-negative lookups. Quick way to confirm: run with `hoodie.metadata.enable=false` and see if dups stop. If so, rebuild the MDT cleanly (remove `<tablePath>/.hoodie/metadata` and let it re-bootstrap), and consider switching the index type (SIMPLE/GLOBAL_SIMPLE, or RECORD_INDEX on 0.14+) to bypass the MDT bloom path. Multiple MDT/index consistency bugs were fixed after 0.13.1, so upgrading to 0.14.x/1.x is the durable fix. For existing dups, use snapshot-query �
�� stage to parquet → delete by key → re-upsert (deduped on max ts), since the CLI dedup is unreliable on MOR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2009520139
@wqwl611 Do you also share the timeline. I see both the records you pasted in screenshot belongs to same file group and both the records are in same file. Couple of questions -
1. Any possibility that the job could have run multi writer jobs without lock configuration?
2. Did you used bulk insert to ingest to this table initially?
3. Do you see any executor failures or retries after that you started seeing dups.
Is this issue occurring frequently now, or only after the upgrade you see this happened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2006164865
can you also share the basic write configs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate data in base file of MOR table [hudi]
Posted by "wqwl611 (via GitHub)" <gi...@apache.org>.
wqwl611 commented on issue #10882:
URL: https://github.com/apache/hudi/issues/10882#issuecomment-2007549709
> If you are using multiwriter, which lock provider you are using? I cant see the lock configuration in the code.
@ad1happy2go
"hoodie.cleaner.policy.failed.writes" -> "LAZY",
"hoodie.write.concurrency.mode" -> "optimistic_concurrency_control",
"hoodie.write.lock.provider" -> "org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider",
"hoodie.write.lock.zookeeper.url" -> zkConnStr,
"hoodie.write.lock.zookeeper.port" -> zkPort,
"hoodie.write.lock.zookeeper.base_path" -> hudiLockBasePath,
"hoodie.write.lock.zookeeper.lock_key" -> tableNameWithSuffix,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org