You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Leoyzen (via GitHub)" <gi...@apache.org> on 2023/02/03 16:54:23 UTC
[GitHub] [hudi] Leoyzen opened a new issue, #7845: [SUPPORT]Unsupported Operation Exception when compacting during flink
Leoyzen opened a new issue, #7845:
URL: https://github.com/apache/hudi/issues/7845
**Describe the problem you faced**
『Unsupported Operation Exception』 raises while compacting.
**To Reproduce**
Steps to reproduce the behavior:
1. compaction is running on MOR table
**Expected behavior**
Compaction successfully.
**Environment Description**
* Hudi version : 0.13.0-rc1
* Spark version : N/A
* Hive version : 3.1.2
* Hadoop version : 3.1.3
* Storage (HDFS/S3/GCS..) : OSS
* Running on Docker? (yes/no) : YES, HA VVP
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```LOG
2023-02-04 00:05:19,130 [pool-26-thread-1] ERROR org.apache.hudi.sink.compact.CompactOperator [] - Executor executes action [Execute compaction for instant 20230201172137801 from task 1] error
org.apache.hudi.exception.HoodieException: Unsupported Operation Exception
at org.apache.hudi.common.util.collection.BitCaskDiskMap.values(BitCaskDiskMap.java:303) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.common.util.collection.ExternalSpillableMap.values(ExternalSpillableMap.java:278) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at java.util.Collections$UnmodifiableMap.values(Collections.java:1487) ~[?:1.8.0_102]
at org.apache.hudi.io.HoodieMergeHandle.writeIncomingRecords(HoodieMergeHandle.java:465) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.io.HoodieMergeHandle.close(HoodieMergeHandle.java:477) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:168) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdateInternal(HoodieFlinkCopyOnWriteTable.java:374) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.table.HoodieFlinkCopyOnWriteTable.handleUpdate(HoodieFlinkCopyOnWriteTable.java:365) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.table.action.compact.CompactionExecutionHelper.writeFileAndGetWriteStats(CompactionExecutionHelper.java:64) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:231) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:144) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.compact.CompactOperator.doCompaction(CompactOperator.java:136) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.compact.CompactOperator.lambda$processElement$0(CompactOperator.java:119) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[flink-connector-hudi-flip-27-1.5-SNAPSHOT-jar-with-dependencies.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147) [?:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) [?:1.8.0_102]
at java.lang.Thread.run(Thread.java:834) [?:1.8.0_102]
```
**Possible Causes**
From the pr https://github.com/apache/hudi/commit/f8028a400ebe7a64d7745fae90e769306a8a07a7， the `getRecords` call from `HoodieMergedLogRecordScanner.java` changed to return a `UnmodifiableMap`.
```JAVA
public Map<String, HoodieRecord> getRecords() {
return Collections.unmodifiableMap(records);
}
```
But in `HoodieMergeHandle`, there is a `instanceof ExternalSpillableMap` which don't meet the condition.
https://github.com/apache/hudi/blob/fc031b9d3b04933aae00eab3c869446fba46b19a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L392
https://github.com/apache/hudi/blob/fc031b9d3b04933aae00eab3c869446fba46b19a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7845: [SUPPORT] Unsupported Operation Exception when compacting during flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #7845:
URL: https://github.com/apache/hudi/issues/7845#issuecomment-1421859209
Close because it is solved, thanks for the report @Leoyzen !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7845: [SUPPORT]Unsupported Operation Exception when compacting during flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #7845:
URL: https://github.com/apache/hudi/issues/7845#issuecomment-1420266608
Thanks, this is a severe regression for release 0.13.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed issue #7845: [SUPPORT] Unsupported Operation Exception when compacting during flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #7845: [SUPPORT] Unsupported Operation Exception when compacting during flink
URL: https://github.com/apache/hudi/issues/7845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org