You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "HSLife1991 (via GitHub)" <gi...@apache.org> on 2024/06/03 11:51:59 UTC
[I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]
HSLife1991 opened a new issue, #6937:
URL: https://github.com/apache/seatunnel/issues/6937
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
when i use deploy seatunnel 2.3.5 with cluster mode, configured hdfs ha for seatunnel.yaml and hazelcast.yaml.
i have put the jar file "seatunnel-hadoop3-3.1.4-uber.jar" in lib directory.
But it reports below error:
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getTimeDuration(Ljava/lang/String;JLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/TimeUnit;)J
at org.apache.hadoop.hdfs.client.impl.DfsClientConf.<init>(DfsClientConf.java:248) 。
### SeaTunnel Version
2.3.5
### SeaTunnel Config
```conf
#seatunnel.yaml
seatunnel:
engine:
history-job-expire-minutes: 1440
backup-count: 1
queue-type: blockingqueue
print-execution-info-interval: 60
print-job-metrics-info-interval: 60
slot-service:
dynamic-slot: true
checkpoint:
interval: 10000
timeout: 60000
storage:
type: hdfs
max-retained: 3
plugin-config:
namespace: /tmp/seatunnel/checkpoint_snapshot
storage.type: hdfs
fs.defaultFS: hdfs://fmxprod # Ensure that the directory has written permission
hdfs_site_path: /usr/hdp/3.1.4.0-315/hadoop/etc/hadoop/hdfs-site.xml
seatunnel.hadoop.dfs.nameservices: fmxprod
seatunnel.hadoop.dfs.ha.namenodes.fmxprod: nn1,nn2
seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn1: bc-gcp-sg-ol-node01.c.proj-bc.internal:8020
seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn2: bc-gcp-sg-ol-node02.c.proj-bc.internal:8020
seatunnel.hadoop.dfs.client.failover.proxy.provider.fmxprod: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
# hazelcast.yaml
hazelcast:
cluster-name: seatunnel
network:
rest-api:
enabled: true
endpoint-groups:
CLUSTER_WRITE:
enabled: true
DATA:
enabled: true
join:
tcp-ip:
enabled: true
member-list:
- bc-gcp-sg-ol-node03.c.proj-bc.internal
- bc-gcp-sg-ol-node04.c.proj-bc.internal
- bc-gcp-sg-ol-node05.c.proj-bc.internal
port:
auto-increment: false
port: 5801
properties:
hazelcast.invocation.max.retry.count: 20
hazelcast.tcp.join.port.try.count: 30
hazelcast.logging.type: log4j2
hazelcast.operation.generic.thread.count: 50
map:
engine*:
map-store:
enabled: true
initial-mode: EAGER
factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
properties:
type: hdfs
namespace: /tmp/seatunnel/imap
clusterName: seatunnel
storage.type: hdfs
fs.defaultFS: hdfs://fmxprod # Ensure that the directory has written permission
hdfs_site_path: /usr/hdp/3.1.4.0-315/hadoop/etc/hadoop/hdfs-site.xml
seatunnel.hadoop.dfs.nameservices: fmxprod
seatunnel.hadoop.dfs.ha.namenodes.fmxprod: nn1,nn2
seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn1: bc-gcp-sg-ol-node01.c.proj-bc.internal:8020
seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn2: bc-gcp-sg-ol-node02.c.proj-bc.internal:8020
seatunnel.hadoop.dfs.client.failover.proxy.provider.fmxprod: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
# setunnelEngine client配置
hazelcast-client:
cluster-name: seatunnel
properties:
hazelcast.logging.type: log4j2
network:
cluster-members:
- bc-gcp-sg-ol-node03.c.proj-bc.internal:5801
- bc-gcp-sg-ol-node04.c.proj-bc.internal:5801
- bc-gcp-sg-ol-node05.c.proj-bc.internal:5801
```
### Running Command
```shell
$SEATUNNEL_HOME/bin/seatunnel-cluster.sh -d
```
### Error Exception
```log
2024-06-03 19:32:19,061 WARN [c.h.s.i.p.ProxyService ] [pool-5-thread-1] - [bc-gcp-sg-ol-node03.c.proj-bc.internal]:5801 [seatunnel] [5.1] Error while initializing proxy: IMap{name='engine_runningJobInfo'}
com.hazelcast.core.HazelcastException: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getTimeDuration(Ljava/lang/String;JLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/TimeUnit;)J
at com.hazelcast.internal.util.ExceptionUtil.lambda$static$0(ExceptionUtil.java:47) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:125) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:115) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:68) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.internal.util.ExceptionUtil.rethrow(ExceptionUtil.java:130) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.map.impl.proxy.MapProxySupport.waitUntilLoaded(MapProxySupport.java:775) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.map.impl.proxy.MapProxyImpl.waitUntilLoaded(MapProxyImpl.java:112) ~[seatunnel-starter.jar:2.3.5]
at com.hazelcast.map.impl.proxy.MapProxySupport.initializeMapStoreLoad(MapProxySupport.java:330) ~[seatunnel-starter.jar:2.3.5]
```
### Zeta or Flink or Spark Version
use default zeta
### Java or Scala Version
java1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6937: [Bug] [Cluster deploy] java.lang.NoSuchMethodError
URL: https://github.com/apache/seatunnel/issues/6937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6937:
URL: https://github.com/apache/seatunnel/issues/6937#issuecomment-2227539665
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6937:
URL: https://github.com/apache/seatunnel/issues/6937#issuecomment-3066303471
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]
Posted by "emmanuelsosa (via GitHub)" <gi...@apache.org>.
emmanuelsosa commented on issue #6937:
URL: https://github.com/apache/seatunnel/issues/6937#issuecomment-2168577873
You might be missing some JARs. Make sure you also have the hadoop jars and the aws java sdk bundle in your $SEATUNNEL_HOME/lib/
```
hadoop-client-3.1.4.jar
hadoop-aws-3.1.4.jar
aws-java-sdk-bundle-1.12.692.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org