You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Atwood1108 (via GitHub)" <gi...@apache.org> on 2024/04/25 06:41:18 UTC
[I] [Bug] [connector-iceberg] Illegal provider-class name: org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.impl.CryptoUtils$Had oopKeyProviderFactory [seatunnel]
Atwood1108 opened a new issue, #6754:
URL: https://github.com/apache/seatunnel/issues/6754
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
connector-iceberg sink使用orc报错
### SeaTunnel Version
2.3.4
### SeaTunnel Config
```conf
env {
job.name = "hive2iceberg_ods_p85_acdoca_di"
job.mode = "BATCH"
# checkpoint.interval = 300000
# checkpoint.timeout = 300000
}
source{
Hive {
parallelism = 40
table_name = "jt_ods.ods_p85_acdoca_di"
metastore_uri = "thrift://ATBGMASTER01:9083,thrift://ATBGMASTER02:9083"
hdfs_site_path = "/data/anta/seatunnel/yzs/config/prod/hadoop-site.xml"
result_table_name = "src"
}
}
transform {
}
sink {
Iceberg {
parallelism = 40
source_table_name = "src"
catalog_name = "seatunnel_yzs"
namespace = "db_yzs"
table = "ods_p85_acdoca_di"
iceberg.hadoop-conf-path = "/data/anta/seatunnel/yzs/config/hbase"
iceberg.catalog.config = {
type = hadoop
warehouse = "hdfs://hadoopcluster:8020/tmp/iceberg"
}
iceberg.table.write-props = {
write.format.default = "orc"
}
schema_save_mode = CREATE_SCHEMA_WHEN_NOT_EXIST
}
}
```
### Running Command
```shell
bin/seatunnel.sh --config hive2iceberg_ods_p85_acdoca_di.conf -e local
```
### Error Exception
```log
2024-04-25 13:42:06,711 ERROR [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-22] - report error from task
org.apache.seatunnel.common.utils.SeaTunnelException: java.util.ServiceConfigurationError: org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.impl.K
eyProvider$Factory: jar:file:/data/anta/seatunnel/connectors/connector-iceberg-2.3.4-2.12.15.jar!/META-INF/services/org.apache.seatunnel.shade.connector-iceb
erg.org.apache.orc.impl.KeyProvider$Factory:15: Illegal provider-class name: org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.impl.CryptoUtils$Had
oopKeyProviderFactory
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.fail(ServiceLoader.java:245)
at java.util.ServiceLoader.parseLine(ServiceLoader.java:272)
at java.util.ServiceLoader.parse(ServiceLoader.java:307)
at java.util.ServiceLoader.access$200(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
at org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.impl.CryptoUtils.getKeyProvider(CryptoUtils.java:140)
at org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.impl.WriterImpl.setupEncryption(WriterImpl.java:953)
at org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:162)
at org.apache.seatunnel.shade.connector-iceberg.org.apache.orc.OrcFile.createWriter(OrcFile.java:1010)
at org.apache.iceberg.orc.ORC.newFileWriter(ORC.java:797)
at org.apache.iceberg.orc.OrcFileAppender.<init>(OrcFileAppender.java:89)
at org.apache.iceberg.orc.ORC$WriteBuilder.build(ORC.java:207)
at org.apache.iceberg.data.GenericAppenderFactory.newAppender(GenericAppenderFactory.java:115)
at org.apache.iceberg.data.GenericAppenderFactory.newDataWriter(GenericAppenderFactory.java:130)
at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.newWriter(BaseTaskWriter.java:352)
at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.newWriter(BaseTaskWriter.java:345)
at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.openCurrent(BaseTaskWriter.java:303)
at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.<init>(BaseTaskWriter.java:265)
at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.<init>(BaseTaskWriter.java:255)
at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.<init>(BaseTaskWriter.java:347)
at org.apache.iceberg.io.UnpartitionedWriter.<init>(UnpartitionedWriter.java:37)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.IcebergWriterFactory.createTaskWriter(IcebergWriterFactory.java:152)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.IcebergRecordWriter.createTaskWriter(IcebergRecordWriter.java:69)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.IcebergRecordWriter.<init>(IcebergRecordWriter.java:65)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.IcebergWriterFactory.createWriter(IcebergWriterFactory.java:89)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.IcebergSinkWriter.tryCreateRecordWriter(IcebergSinkWriter.java:96)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.IcebergSinkWriter.<init>(IcebergSinkWriter.java:77)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.IcebergSinkWriter.of(IcebergSinkWriter.java:109)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.IcebergSinkWriter.of(IcebergSinkWriter.java:101)
at org.apache.seatunnel.connectors.seatunnel.iceberg.sink.IcebergSink.createWriter(IcebergSink.java:93)
at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSink.createWriter(MultiTableSink.java:70)
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:297)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$16(SeaTunnelTask.java:425)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:422)
at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:372) [seatunnel-start
er.jar:2.3.4]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:183) [seatunnel-starter.jar:2
.3.4]
at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.run(CheckpointErrorReportOperation.java:48) [seatunnel-star
ter.jar:2.3.4]
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) [seatunnel-starter.jar:2.3.4]
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.4]
```
### Zeta or Flink or Spark Version
zeta 2.3.4
### Java or Scala Version
java 1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org