You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Vinoth Chandar (Jira)" <ji...@apache.org> on 2019/10/21 00:38:00 UTC
[jira] [Closed] (HUDI-174) Investigate atomicity guarantees out of
cloud storage
[ https://issues.apache.org/jira/browse/HUDI-174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinoth Chandar closed HUDI-174.
-------------------------------
Resolution: Fixed
> Investigate atomicity guarantees out of cloud storage
> -----------------------------------------------------
>
> Key: HUDI-174
> URL: https://issues.apache.org/jira/browse/HUDI-174
> Project: Apache Hudi (incubating)
> Issue Type: Task
> Components: Write Client
> Reporter: Vinoth Chandar
> Assignee: Balaji Varadarajan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Bug report :
> we are getting a "File Id has more than 1 pending compaction" error. How would I go about resolving this? (still learning hudi). Here is the stack trace:
> {code}
> com.facebook.presto.spi.PrestoException: Hoodie File Id (HoodieFileGroupId{partitionPath='2019/07/17', fileId='ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0'}) has more thant 1 pending compactions. Instants: (20190718213318,{"baseInstantTime": "20190718212749", "deltaFilePaths": ["gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/.ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_20190718212749.log.1_1-21-4523"], "dataFilePath": "gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_0-35-4549_20190718212749.parquet", "fileId": "ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0", "partitionPath": "2019\/07\/17", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 3.0, "TOTAL_LOG_FILES_SIZE": 84287.0, "TOTAL_IO_WRITE_MB": 3.0, "TOTAL_IO_MB": 6.0, "TOTAL_LOG_FILE_SIZE": 84287.0}}), (20190718213318,{"baseInstantTime": "20190718212749", "deltaFilePaths": ["gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/.ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_20190718212749.log.1_1-21-4523"], "dataFilePath": "gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_0-35-4549_20190718212749.parquet", "fileId": "ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0", "partitionPath": "2019\/07\/17", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 3.0, "TOTAL_LOG_FILES_SIZE": 84287.0, "TOTAL_IO_WRITE_MB": 3.0, "TOTAL_IO_MB": 6.0, "TOTAL_LOG_FILE_SIZE": 84287.0}})
> at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:192)
> at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
> at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
> at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
> at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Hoodie File Id (HoodieFileGroupId{partitionPath='2019/07/17', fileId='ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0'}) has more thant 1 pending compactions. Instants: (20190718213318,{"baseInstantTime": "20190718212749", "deltaFilePaths": ["gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/.ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_20190718212749.log.1_1-21-4523"], "dataFilePath": "gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_0-35-4549_20190718212749.parquet", "fileId": "ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0", "partitionPath": "2019\/07\/17", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 3.0, "TOTAL_LOG_FILES_SIZE": 84287.0, "TOTAL_IO_WRITE_MB": 3.0, "TOTAL_IO_MB": 6.0, "TOTAL_LOG_FILE_SIZE": 84287.0}}), (20190718213318,{"baseInstantTime": "20190718212749", "deltaFilePaths": ["gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/.ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_20190718212749.log.1_1-21-4523"], "dataFilePath": "gs:\/\/hudi-ingest\/hudi\/data\/hudi_ingest_raw\/updates_latest\/2019\/07\/17\/ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0_0-35-4549_20190718212749.parquet", "fileId": "ba820041-3e16-4fbd-b0a0-9e8ad22ade82-0", "partitionPath": "2019\/07\/17", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 3.0, "TOTAL_LOG_FILES_SIZE": 84287.0, "TOTAL_IO_WRITE_MB": 3.0, "TOTAL_IO_MB": 6.0, "TOTAL_LOG_FILE_SIZE": 84287.0}})
> at com.uber.hoodie.common.util.CompactionUtils.lambda$getAllPendingCompactionOperations$5(CompactionUtils.java:158)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)