You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "lordk911 (via GitHub)" <gi...@apache.org> on 2023/03/30 09:19:25 UTC
[GitHub] [incubator-seatunnel] lordk911 opened a new issue, #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
lordk911 opened a new issue, #4454:
URL: https://github.com/apache/incubator-seatunnel/issues/4454
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Follow the quick start document，run `./bin/seatunnel.sh --config ./config/v2.streaming.conf.template -e local` will get java.lang.NoClassDefFoundError
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
nothing special
```
### Running Command
```shell
./bin/seatunnel.sh --config ./config/v2.streaming.conf.template -e local
```
### Error Exception
```log
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251)
at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243)
at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245)
at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135)
at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53)
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:75)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:101)
at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:240)
at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:223)
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:417)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] yiyunlei commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "yiyunlei (via GitHub)" <gi...@apache.org>.
yiyunlei commented on issue #4454:
URL: https://github.com/apache/seatunnel/issues/4454#issuecomment-1571158215
I also encountered this bug. The solution provided here works for me too. Thanks for bringing this up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] caoyongfly commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "caoyongfly (via GitHub)" <gi...@apache.org>.
caoyongfly commented on issue #4454:
URL: https://github.com/apache/incubator-seatunnel/issues/4454#issuecomment-1556066762
I meet this problem too,download this jar and put it into the path /seatunnel-2.3.1/lib can fix the problem
https://repo.maven.apache.org/maven2/org/apache/seatunnel/seatunnel-hadoop3-3.1.4-uber/2.3.1/seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lordk911 commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "lordk911 (via GitHub)" <gi...@apache.org>.
lordk911 commented on issue #4454:
URL: https://github.com/apache/incubator-seatunnel/issues/4454#issuecomment-1489979259
have to add some jars to the lib directory：
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] Camellia-lili commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "Camellia-lili (via GitHub)" <gi...@apache.org>.
Camellia-lili commented on issue #4454:
URL: https://github.com/apache/seatunnel/issues/4454#issuecomment-1560398609
Yes, Me too.I also meet this question,I also download a jar(seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar) ,and put it into the path of seatunnel-2.3.1/lib，perfectly solved this question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lordk911 closed issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "lordk911 (via GitHub)" <gi...@apache.org>.
lordk911 closed issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
URL: https://github.com/apache/incubator-seatunnel/issues/4454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lordk911 commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "lordk911 (via GitHub)" <gi...@apache.org>.
lordk911 commented on issue #4454:
URL: https://github.com/apache/incubator-seatunnel/issues/4454#issuecomment-1489995296
maybe this is needed:
`https://repo.maven.apache.org/maven2/org/apache/seatunnel/seatunnel-hadoop3-3.1.4-uber/2.3.1/seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] laglangyue commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4454:
URL: https://github.com/apache/incubator-seatunnel/issues/4454#issuecomment-1490447755
We have not explicitly adapted 3.0,3.1, 3.2, can you use 3.3
And I will test it on the weekend for 3.0, 3.1,3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lordk911 commented on issue #4454: [Bug] [seatunnel-engine/seatunnel-engine-storage] faild to start under HDP3.1.4(hadoop3.1) Environment
Posted by "lordk911 (via GitHub)" <gi...@apache.org>.
lordk911 commented on issue #4454:
URL: https://github.com/apache/incubator-seatunnel/issues/4454#issuecomment-1491166889
> maybe this is needed: `https://repo.maven.apache.org/maven2/org/apache/seatunnel/seatunnel-hadoop3-3.1.4-uber/2.3.1/seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar`
This can be use to adapte hadoop3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org