You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "ccczhouxin (via GitHub)" <gi...@apache.org> on 2024/07/08 03:24:05 UTC

[I] The seatunnel cluster has some error [seatunnel]

ccczhouxin opened a new issue, #7127:
URL: https://github.com/apache/seatunnel/issues/7127

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   Sending data to kafka is an error,the cluster has two nodes,one has offline
   
   
   ### SeaTunnel Version
   
   2.3.5
   
   ### SeaTunnel Config
   
   ```conf
   hazelcast:
     cluster-name: seatunnel-xugu
     network:
       rest-api:
         enabled: true
         endpoint-groups:
           CLUSTER_WRITE:
             enabled: true
           DATA:
             enabled: true
       join:
         tcp-ip:
           enabled: true
           member-list:
             - a:5801
             - b:5801
       port:
         auto-increment: true
         port: 5801
     properties:
       hazelcast.invocation.max.retry.count: 20
       hazelcast.tcp.join.port.try.count: 30
       hazelcast.logging.type: log4j2
       hazelcast.operation.generic.thread.count: 50
   
   
   hazelcast-client:
     cluster-name: seatunnel-xugu
     properties:
       hazelcast.logging.type: log4j2
     connection-strategy:
       connection-retry:
         cluster-connect-timeout-millis: 3000
     network:
       cluster-members:
         - a:5801
         - b:5801
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh --config config/a
   ```
   
   
   ### Error Exception
   
   ```log
   2024-07-07 07:56:36,690 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-xugu] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
   2024-07-07 07:56:36,734 WARN  [.c.i.s.ClientInvocationService] [hz.client_1.internal-4] - hz.client_1 [seatunnel-xugu] [5.1] Future.complete(Object) on completed future. Request: ClientMessage{connection=null, length=119, operation=Client.Authentication, isRetryable=true, correlationId=19899, messageType=100, isEvent=false, isfragmented=false}, current value: ExceptionalResult{cause=java.lang.InterruptedException}, offered value: ExceptionalResult{cause=com.hazelcast.client.HazelcastClientNotActiveException: Client is shutting down}
   java.lang.Exception: null
     at com.hazelcast.spi.impl.AbstractInvocationFuture.warnIfSuspiciousDoubleCompletion(AbstractInvocationFuture.java:1253) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1230) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:316) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:758) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:191) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.authenticateOnCluster(TcpClientConnectionManager.java:885) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToMember(TcpClientConnectionManager.java:627) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager$ConnectToAllClusterMembersTask.lambda$run$0(TcpClientConnectionManager.java:1182) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_221]
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_221]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_221]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_221]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
     at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [seatunnel-starter.jar:2.3.6-SNAPSHOT]
     at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.6-SNAPSHOT]
   2024-07-07 07:56:36,737 WARN  [.c.i.c.ClientConnectionManager] [hz.client_1.internal-4] - hz.client_1 [seatunnel-xugu] [5.1] Could not connect to member eafe4bcc-2b7e-47be-9ef9-dea9fed61f42, reason com.hazelcast.core.HazelcastException: java.lang.InterruptedException
   2024-07-07 07:56:36,739 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-xugu] [5.1] Removed connection to endpoint: [localhost]:5801:0db5a450-6149-480f-ba92-345b8eb1bca9, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/localhost:28092->/localhost:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-07-07 07:56:36.676, lastWriteTime=2024-07-07 07:56:36.238, closedTime=2024-07-07 07:56:36.737, connected server version=5.1}
   2024-07-07 07:56:36,739 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-xugu] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
   2024-07-07 07:56:36,743 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-xugu] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
   2024-07-07 07:56:36,743 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
   2024-07-07 07:56:36,743 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
   2024-07-07 07:56:36,743 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -
   
   ===============================================================================
   
   
   2024-07-07 07:56:36,743 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,
   
   2024-07-07 07:56:36,743 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues
   
   2024-07-07 07:56:36,744 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed
   
   2024-07-07 07:56:36,746 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
     at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:209)
     at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
     at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.common.exception.JobException: The taskGroup(TaskGroupLocation{jobId=1464550058967051, pipelineId=1, taskGroupId=1}) deployed node([slave]:5801) offline
     at org.apache.seatunnel.engine.server.CoordinatorService.lambda$makeTasksFailed$13(CoordinatorService.java:729)
     at java.util.ArrayList.forEach(ArrayList.java:1257)
     at org.apache.seatunnel.engine.server.CoordinatorService.makeTasksFailed(CoordinatorService.java:714)
     at org.apache.seatunnel.engine.server.CoordinatorService.lambda$null$11(CoordinatorService.java:704)
     at java.util.ArrayList.forEach(ArrayList.java:1257)
     at org.apache.seatunnel.engine.server.CoordinatorService.lambda$failedTaskOnMemberRemoved$12(CoordinatorService.java:702)
     at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)
     at org.apache.seatunnel.engine.server.CoordinatorService.failedTaskOnMemberRemoved(CoordinatorService.java:697)
     at org.apache.seatunnel.engine.server.CoordinatorService.memberRemoved(CoordinatorService.java:740)
     at org.apache.seatunnel.engine.server.SeaTunnelServer.memberRemoved(SeaTunnelServer.java:162)
     at com.hazelcast.internal.cluster.impl.MembershipManager.lambda$sendMembershipEventNotifications$2(MembershipManager.java:835)
     at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:748)
     at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
     at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
   
     at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:201)
     ... 2 more
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   1.8
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] The seatunnel cluster has some error [seatunnel]

Posted by "ccczhouxin (via GitHub)" <gi...@apache.org>.
ccczhouxin commented on issue #7127:
URL: https://github.com/apache/seatunnel/issues/7127#issuecomment-2216105163

   thanks 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] The seatunnel cluster has some error [seatunnel]

Posted by "emmanuelsosa (via GitHub)" <gi...@apache.org>.
emmanuelsosa commented on issue #7127:
URL: https://github.com/apache/seatunnel/issues/7127#issuecomment-2215112215

   To ensure fault tolerance, your cluster must consist of at least three members. In hazelcast clusters that involve leader elections, you need maintain an odd number of nodes (and not just one) to facilitate an undisputed leader election.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] The seatunnel cluster has some error [seatunnel]

Posted by "ccczhouxin (via GitHub)" <gi...@apache.org>.
ccczhouxin closed issue #7127: The seatunnel cluster has some error 
URL: https://github.com/apache/seatunnel/issues/7127


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org