You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2021/12/10 16:13:00 UTC
[jira] [Resolved] (HUDI-2952) Metadata table compaction fails for non partitioned dataset
[ https://issues.apache.org/jira/browse/HUDI-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan resolved HUDI-2952.
---------------------------------------
> Metadata table compaction fails for non partitioned dataset
> -----------------------------------------------------------
>
> Key: HUDI-2952
> URL: https://issues.apache.org/jira/browse/HUDI-2952
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Writer Core
> Affects Versions: 0.10.0
> Reporter: sivabalan narayanan
> Assignee: Manoj Govindassamy
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Looks like the conversion of empty partition path to "." (dot) is not fixed in all places when we convert the records for metadata table. like clean metadata and rollback metadata. [link|https://github.com/apache/hudi/blob/e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L133].
>
> {code:java}
> Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old record into new file for
> key from old file s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
> tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile to new file s3a://testing-4
> 27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
> s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
> "type" : "record",
> "name" : "HoodieMetadataRecord",
> "namespace" : "org.apache.hudi.avro.model",
> "doc" : "A record saved within the Metadata Table",
> "fields" : [ {
> "name" : "_hoodie_commit_time",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> "name" : "_hoodie_commit_seqno",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> "name" : "_hoodie_record_key",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> .
> .
> .
> at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
> at org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
> at org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
> at org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
> at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalArgumentException: key length must be > 0
> at org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
> at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
> at org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
> at org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
> at org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
> at org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
> ... 9 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)