You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/28 09:45:31 UTC
[GitHub] [hudi] Chosen123Wang opened a new issue, #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Chosen123Wang opened a new issue, #5457:
URL: https://github.com/apache/hudi/issues/5457
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.11.0
* Spark version :3.2.1
* Hive version :3.1.2
* Hadoop version : 3.2.2
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
2022-04-28 17:19:05,955 INFO org.apache.hadoop.io.compress.CodecPool [] - Got brand-new compressor [.gz]
2022-04-28 17:19:05,955 INFO org.apache.flink.streaming.runtime.tasks.RegularOperatorChain [] - Could not complete snapshot 66 for operator stream_write -> Sink: clean_commits (1/1)#63. Failure reason: Checkpoint was declined.
org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 66 for operator stream_write -> Sink: clean_commits (1/1)#63. Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) [flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) [flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) [flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist_2.12-1.14.3.jar:1.14.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.ClassCastException: org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
at org.apache.hudi.io.storage.HoodieParquetWriter.<init>(HoodieParquetWriter.java:72) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:84) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:70) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:54) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:101) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:80) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.FlinkCreateHandle.<init>(FlinkCreateHandle.java:67) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.FlinkCreateHandle.<init>(FlinkCreateHandle.java:60) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:479) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:143) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:184) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608) ~[?:1.8.0_282]
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
... 33 more
2022-04-28 17:19:05,956 INFO org.apache.hudi.async.AsyncCleanerService [] - Shutting down async clean service...
2022-04-28 17:19:05,956 WARN org.apache.flink.runtime.taskmanager.Task [] - stream_write -> Sink: clean_commits (1/1)#63 (9a1c1c788f0890ec8536895ab4d266c0) switched from RUNNING to FAILED with failure cause: java.io.IOException: Could not perform checkpoint 66 for operator stream_write -> Sink: clean_commits (1/1)#63.
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1274)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493)
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 66 for operator stream_write -> Sink: clean_commits (1/1)#63. Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258)
... 22 more
Caused by: java.lang.ClassCastException: org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
at org.apache.hudi.io.storage.HoodieParquetWriter.<init>(HoodieParquetWriter.java:72)
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:84)
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:70)
at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:54)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:101)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:80)
at org.apache.hudi.io.FlinkCreateHandle.<init>(FlinkCreateHandle.java:67)
at org.apache.hudi.io.FlinkCreateHandle.<init>(FlinkCreateHandle.java:60)
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:479)
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:143)
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:184)
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454)
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219)
... 33 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1112368428
@Chosen123Wang To understand the issue, could you share the Hudi write configs and the steps and commands to reproduce the problem? It looks like you're using Flink instead of Spark to write the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1120185360
> @Chosen123Wang Could you try this fix to see if it solves your issue: #5471?
In my tests, some jobs can succeed and others can't. I would like to ask which class loader is loaded by the houdiewrapperfilesystem class. We want to specify it directly in the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1469453657
It's not that easy to resolve, see a detaied analysis from Ken: https://github.com/apache/hudi/issues/8148, kind of flink classloader + hadoop fs bug here, what mode do you use for flink, the session cluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1125703125
It seems in latest master code, we already comment out the variable in `HoodieBaseParquetWriter` ? cc @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang closed issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang closed issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
URL: https://github.com/apache/hudi/issues/5457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] langyalzy commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
langyalzy commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1125585654
Hi, I also encountered the same problem. My approach here is
1. Modify the code in org.apache.hudi.io.storage.HoodieParquetWriter of hudi-client-common:
**Comment out this variable directly HoodieWrapperFileSystem fs**
[Before]
private final HoodieWrapperFileSystem fs;
[after]
//private final HoodieWrapperFileSystem fs;
2. Continue to modify the above 'fs' variable association method code in this class:
**Comment out the 'fs' object reference code in the constructor of org.apache.hudi.io.storage.HoodieParquetWriter directly**
[Before]
this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
[after]
//this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
Because this variable is not used for the time being, commenting it out will solve my problem, you can also try this method, I hope it can help you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1113629138
@Chosen123Wang After some investigation, this might be related to the class loading configuration in Java, as @YuangZhang pointed out. I'll put up a patch for you to try out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1169641607
@Chosen123Wang I need to check the usage. If it's not used, we should remove it. The writer is used on the executor side so likely that HoodieWrapperFileSystem is somehow not on the classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang closed issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang closed issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
URL: https://github.com/apache/hudi/issues/5457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] c-f-cooper commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1469333663
hi, The problem also appeared in my work,I use flink 1.15,hudi0.13.2,cloud you help me ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] c-f-cooper commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1469667841
> > >
> >
> >
> > application
>
> What classloader did you config? child-first or parent-first ?
parent-first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1652825409
+1, Using flink bulk_insert action, this exception still happen in Hudi 0.13.1 + flink1.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1125586398
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1113885251
@Chosen123Wang Could you try this fix to see if it solves your issue: #5471?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] c-f-cooper commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1469464588
>
application
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1652843222
Still catch this exception in Hudi 0.13.1 + flink 1.14 using bulk_insert operation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1125690477
> Hi, I also encountered the same problem. My approach here is
>
> 1. Modify the code in org.apache.hudi.io.storage.HoodieParquetWriter of hudi-client-common:
> **Comment out this variable directly HoodieWrapperFileSystem fs**
> [Before]
> private final HoodieWrapperFileSystem fs;
> [after]
> //private final HoodieWrapperFileSystem fs;
> 2. Continue to modify the above 'fs' variable association method code in this class:
>
> **Comment out the 'fs' object reference code in the constructor of org.apache.hudi.io.storage.HoodieParquetWriter directly** [Before] this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf())); [after] //this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
>
> Because this variable is not used for the time being, commenting it out will solve my problem, you can also try this method, I hope it can help you.
According to your solution, I tested it, the job is running normally, thank you for solving my problem。
What does this variable do? @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuangZhang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
YuangZhang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1112814459
I have encountered such errors，this is because different classloader loads a same class，you need check you library
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1652955532
Thanks for the feedback @zhangyue19921010 , how you solve this issue finally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1764051979
Should be fixed via: https://github.com/apache/hudi/pull/9868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1469666372
> >
>
> application
What classloader did you config? child-first or parent-first ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1652996762
Yeap, Will raise a PR to fix this issue today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1119420615
> @Chosen123Wang Could you try this fix to see if it solves your issue: #5471?
I added this line of code ,and still report this error
hudi 0.11.0-release flink 1.14.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1119450103
> I have encountered such errors，this is because different classloader loads a same class，you need check you library
What dependencies do I need to check ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Chosen123Wang commented on issue #5457: [SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
Posted by GitBox <gi...@apache.org>.
Chosen123Wang commented on issue #5457:
URL: https://github.com/apache/hudi/issues/5457#issuecomment-1125586976
> Hi, I also encountered the same problem. My approach here is
>
> 1. Modify the code in org.apache.hudi.io.storage.HoodieParquetWriter of hudi-client-common:
> **Comment out this variable directly HoodieWrapperFileSystem fs**
> [Before]
> private final HoodieWrapperFileSystem fs;
> [after]
> //private final HoodieWrapperFileSystem fs;
> 2. Continue to modify the above 'fs' variable association method code in this class:
>
> **Comment out the 'fs' object reference code in the constructor of org.apache.hudi.io.storage.HoodieParquetWriter directly** [Before] this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf())); [after] //this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
>
> Because this variable is not used for the time being, commenting it out will solve my problem, you can also try this method, I hope it can help you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org