You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/17 13:01:31 UTC
[GitHub] [hudi] Guanpx opened a new issue #5061: [SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）
Guanpx opened a new issue #5061:
URL: https://github.com/apache/hudi/issues/5061
**Describe the problem you faced**
USE BUCKET index， when flink chkp that throw Exception **java.util.NoSuchElementException: No value present in Option**
A clear and concise description of the problem.
**To Reproduce**
BUCKET index, COW table , insert op
0. chkp
1. hudi config
```
'connector' = 'hudi',
'path' = 'hdfs://xxxx',
'index.bootstrap.enabled' = 'false',
'index.state.ttl' = '0.0',
'index.type' = 'BUCKET',
'hoodie.parquet.compression.codec'= 'snappy',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'insert',
'write.precombine' = 'true',
'write.precombine.field' = 'ts',
'write.tasks' = '1',
'write.bucket_assign.tasks' = '1',
'hive_sync.enable'='false',
'hive_sync.table'='xxxx',
'hive_sync.db'='xxx',
'hive_sync.mode' = 'hms',
'hive_sync.metastore.uris' = 'thrift://xxxx.com:9083',
'hive_sync.skip_ro_suffix' = 'true',
'write.insert.cluster' = 'true',
'write.ignore.failed' = 'true',
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : master 0.11
* Flink version : 1.14.3
* Hadoop version : 3.0.0-cdh6
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
2022-03-17 20:38:31
java.io.IOException: Could not perform checkpoint 3 for operator bucket_write -> Sink: clean_commits (1/1)#2.
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1274)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493)
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 3 for operator bucket_write -> Sink: clean_commits (1/1)#2. Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258)
... 22 more
Caused by: java.util.NoSuchElementException: No value present in Option
at org.apache.hudi.common.util.Option.get(Option.java:88)
at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:118)
at org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:70)
at org.apache.hudi.io.FlinkConcatHandle.<init>(FlinkConcatHandle.java:53)
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:480)
at org.apache.hudi.client.HoodieFlinkWriteClient.insert(HoodieFlinkWriteClient.java:170)
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$0(StreamWriteFunction.java:181)
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454)
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219)
... 33 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wxplovecc commented on issue #5061: [SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）
Posted by GitBox <gi...@apache.org>.
wxplovecc commented on issue #5061:
URL: https://github.com/apache/hudi/issues/5061#issuecomment-1072220081
see https://github.com/apache/hudi/pull/5018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Guanpx commented on issue #5061: [SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）
Posted by GitBox <gi...@apache.org>.
Guanpx commented on issue #5061:
URL: https://github.com/apache/hudi/issues/5061#issuecomment-1072274424
> see #5018
thx, It looks good, and run without Exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wxplovecc commented on issue #5061: [SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）
Posted by GitBox <gi...@apache.org>.
wxplovecc commented on issue #5061:
URL: https://github.com/apache/hudi/issues/5061#issuecomment-1072220081
see https://github.com/apache/hudi/pull/5018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Guanpx closed issue #5061: [SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）
Posted by GitBox <gi...@apache.org>.
Guanpx closed issue #5061:
URL: https://github.com/apache/hudi/issues/5061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Guanpx commented on issue #5061: [SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）
Posted by GitBox <gi...@apache.org>.
Guanpx commented on issue #5061:
URL: https://github.com/apache/hudi/issues/5061#issuecomment-1072274424
> see #5018
thx, It looks good, and run without Exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org