You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "HSLife1991 (via GitHub)" <gi...@apache.org> on 2024/06/03 11:51:59 UTC

[I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]

HSLife1991 opened a new issue, #6937:
URL: https://github.com/apache/seatunnel/issues/6937

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   when i use deploy seatunnel 2.3.5 with cluster mode, configured hdfs ha for seatunnel.yaml and hazelcast.yaml.
   i have put the jar file "seatunnel-hadoop3-3.1.4-uber.jar" in lib directory.
    
   But it reports below error:
   Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getTimeDuration(Ljava/lang/String;JLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/TimeUnit;)J
           at org.apache.hadoop.hdfs.client.impl.DfsClientConf.<init>(DfsClientConf.java:248) 。
   
   
   ### SeaTunnel Version
   
   2.3.5
   
   ### SeaTunnel Config
   
   ```conf
   #seatunnel.yaml
   seatunnel:
     engine:
       history-job-expire-minutes: 1440
       backup-count: 1
       queue-type: blockingqueue
       print-execution-info-interval: 60
       print-job-metrics-info-interval: 60
       slot-service:
         dynamic-slot: true
       checkpoint:
         interval: 10000
         timeout: 60000
         storage:
           type: hdfs
           max-retained: 3
           plugin-config:
             namespace: /tmp/seatunnel/checkpoint_snapshot
             storage.type: hdfs
             fs.defaultFS: hdfs://fmxprod # Ensure that the directory has written permission
             hdfs_site_path: /usr/hdp/3.1.4.0-315/hadoop/etc/hadoop/hdfs-site.xml
             seatunnel.hadoop.dfs.nameservices: fmxprod
             seatunnel.hadoop.dfs.ha.namenodes.fmxprod: nn1,nn2
             seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn1: bc-gcp-sg-ol-node01.c.proj-bc.internal:8020
             seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn2: bc-gcp-sg-ol-node02.c.proj-bc.internal:8020
             seatunnel.hadoop.dfs.client.failover.proxy.provider.fmxprod: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
   
   
   
   # hazelcast.yaml
   hazelcast:
     cluster-name: seatunnel
     network:
       rest-api:
         enabled: true
         endpoint-groups:
           CLUSTER_WRITE:
             enabled: true
           DATA:
             enabled: true
       join:
         tcp-ip:
           enabled: true
           member-list:
             - bc-gcp-sg-ol-node03.c.proj-bc.internal
             - bc-gcp-sg-ol-node04.c.proj-bc.internal
             - bc-gcp-sg-ol-node05.c.proj-bc.internal
       port:
         auto-increment: false
         port: 5801
     properties:
       hazelcast.invocation.max.retry.count: 20
       hazelcast.tcp.join.port.try.count: 30
       hazelcast.logging.type: log4j2
       hazelcast.operation.generic.thread.count: 50
     map:
       engine*:
          map-store:
            enabled: true
            initial-mode: EAGER
            factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
            properties:
              type: hdfs
              namespace: /tmp/seatunnel/imap
              clusterName: seatunnel
              storage.type: hdfs
              fs.defaultFS: hdfs://fmxprod # Ensure that the directory has written permission
              hdfs_site_path: /usr/hdp/3.1.4.0-315/hadoop/etc/hadoop/hdfs-site.xml
              seatunnel.hadoop.dfs.nameservices: fmxprod
              seatunnel.hadoop.dfs.ha.namenodes.fmxprod: nn1,nn2
              seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn1: bc-gcp-sg-ol-node01.c.proj-bc.internal:8020
              seatunnel.hadoop.dfs.namenode.rpc-address.fmxprod.nn2: bc-gcp-sg-ol-node02.c.proj-bc.internal:8020
              seatunnel.hadoop.dfs.client.failover.proxy.provider.fmxprod: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
   
   
   # setunnelEngine client配置
   hazelcast-client:
     cluster-name: seatunnel
     properties:
       hazelcast.logging.type: log4j2
     network:
       cluster-members:
         - bc-gcp-sg-ol-node03.c.proj-bc.internal:5801
         - bc-gcp-sg-ol-node04.c.proj-bc.internal:5801
         - bc-gcp-sg-ol-node05.c.proj-bc.internal:5801
   ```
   
   
   ### Running Command
   
   ```shell
   $SEATUNNEL_HOME/bin/seatunnel-cluster.sh -d
   ```
   
   
   ### Error Exception
   
   ```log
   2024-06-03 19:32:19,061 WARN  [c.h.s.i.p.ProxyService        ] [pool-5-thread-1] - [bc-gcp-sg-ol-node03.c.proj-bc.internal]:5801 [seatunnel] [5.1] Error while initializing proxy: IMap{name='engine_runningJobInfo'}
   com.hazelcast.core.HazelcastException: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getTimeDuration(Ljava/lang/String;JLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/TimeUnit;)J
           at com.hazelcast.internal.util.ExceptionUtil.lambda$static$0(ExceptionUtil.java:47) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:125) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:115) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:68) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.internal.util.ExceptionUtil.rethrow(ExceptionUtil.java:130) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.map.impl.proxy.MapProxySupport.waitUntilLoaded(MapProxySupport.java:775) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.map.impl.proxy.MapProxyImpl.waitUntilLoaded(MapProxyImpl.java:112) ~[seatunnel-starter.jar:2.3.5]
           at com.hazelcast.map.impl.proxy.MapProxySupport.initializeMapStoreLoad(MapProxySupport.java:330) ~[seatunnel-starter.jar:2.3.5]
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   use default zeta
   
   ### Java or Scala Version
   
   java1.8
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6937: [Bug] [Cluster deploy] java.lang.NoSuchMethodError
URL: https://github.com/apache/seatunnel/issues/6937


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6937:
URL: https://github.com/apache/seatunnel/issues/6937#issuecomment-2227539665

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6937:
URL: https://github.com/apache/seatunnel/issues/6937#issuecomment-3066303471

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Cluster deploy] java.lang.NoSuchMethodError [seatunnel]

Posted by "emmanuelsosa (via GitHub)" <gi...@apache.org>.
emmanuelsosa commented on issue #6937:
URL: https://github.com/apache/seatunnel/issues/6937#issuecomment-2168577873

   You might be missing some JARs. Make sure you also have the hadoop jars and the aws java sdk bundle in your $SEATUNNEL_HOME/lib/
   
   ```
   hadoop-client-3.1.4.jar
   hadoop-aws-3.1.4.jar
   aws-java-sdk-bundle-1.12.692.jar
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org