You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@seatunnel.apache.org by "cqutwangyu (via GitHub)" <gi...@apache.org> on 2024/02/04 06:14:18 UTC

[I] [Bug] [Zeta] OutOfMemoryError: Compressed class space [seatunnel]

cqutwangyu opened a new issue, #6333:
URL: https://github.com/apache/seatunnel/issues/6333

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   I've seen similar problems but they haven't been solved. https://github.com/apache/seatunnel/issues/5442
   
   I started a SeaTunnelServer process via seatunnel-cluster.sh.
   
   I ran the seatunnel streaming task for RocketMQ message forwarding in cluster mode in dolphinscheduler (DS).
   
   In DS, I set this task to be scheduled every minute, and the execution strategy is serial abandonment, so as to realize fast recovery after the seatunnel flow task fails.
   
   RocketMQ was out of service for some time during the run of the scheduled task, while DS continued to pull seatunnel once every minute for a longer time.
   
   After that, RocketMQ resumes normal service, but running any seatunnel task at this time will result in an OOM exception.
   
   ### SeaTunnel Version
   
   2.3.3-release
   
   ### SeaTunnel Config
   
   ```conf
   env {
       parallelism = 4
       job.mode = "STREAMING"
   }
   
   source {
     Rocketmq {
       name.srv.addr = "127.0.0.1:9876"
       topics = "seatunnel_source"
     }
   }
   transform {
   }
   
   sink {
     Rocketmq {
       name.srv.addr = "127.0.0.1:9876"
       topic = "seatunnel_sink"
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   sh bin/seatunnel.sh --config config/v2.batch.config.template -m cluster
   ```
   
   
   ### Error Exception
   
   ```log
   Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/opt/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
   Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /opt/apache-seatunnel-2.3.3/config/seatunnel.yaml
   Feb 04, 2024 1:51:42 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
   INFO: seatunnel.home is /opt/apache-seatunnel-2.3.3
   Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/opt/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
   Feb 04, 2024 1:51:42 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /opt/apache-seatunnel-2.3.3/config/hazelcast.yaml
   Feb 04, 2024 1:51:43 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/opt/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
   Feb 04, 2024 1:51:43 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /opt/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
   2024-02-04 13:51:43,544 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
   2024-02-04 13:51:43,612 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
   2024-02-04 13:51:43,617 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
   2024-02-04 13:51:43,644 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
   2024-02-04 13:51:43,648 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
   2024-02-04 13:51:43,683 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
   2024-02-04 13:51:43,683 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:382638e7-e321-4675-acea-247579799b4b, server version: 5.1, local address: /127.0.0.1:49198
   2024-02-04 13:51:43,685 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
   2024-02-04 13:51:43,705 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1]
   
   Members [1] {
           Member [localhost]:5801 - 382638e7-e321-4675-acea-247579799b4b
   }
   
   2024-02-04 13:51:43,730 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
   2024-02-04 13:51:43,879 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
   2024-02-04 13:51:43,890 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: config/v2.batch.config.template
   2024-02-04 13:51:43,956 INFO  org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
   2024-02-04 13:51:43,999 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
       "env" : {
           "execution.parallelism" : 2,
           "job.mode" : "BATCH",
           "checkpoint.interval" : 10000
       },
       "source" : [
           {
               "schema" : {
                   "fields" : {
                       "name" : "string",
                       "age" : "int"
                   }
               },
               "row.num" : 16,
               "parallelism" : 2,
               "result_table_name" : "fake",
               "plugin_name" : "FakeSource"
           }
       ],
       "sink" : [
           {
               "plugin_name" : "Console"
           }
       ]
   }
   
   2024-02-04 13:51:44,024 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,025 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,029 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /opt/apache-seatunnel-2.3.3/connectors/seatunnel
   2024-02-04 13:51:44,035 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
   2024-02-04 13:51:44,036 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
   2024-02-04 13:51:44,039 INFO  org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
   2024-02-04 13:51:44,039 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,039 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,039 WARN  org.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option.
   2024-02-04 13:51:44,040 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
   2024-02-04 13:51:44,041 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,094 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /opt/apache-seatunnel-2.3.3/connectors/seatunnel
   2024-02-04 13:51:44,099 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
   2024-02-04 13:51:44,101 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath
   2024-02-04 13:51:44,121 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms.
   2024-02-04 13:51:44,122 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks.
   2024-02-04 13:51:44,122 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,128 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-02-04 13:51:44,169 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 806401517069271041, with plugin jar [file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/opt/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
   2024-02-04 13:51:49,218 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
   2024-02-04 13:51:49,221 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:382638e7-e321-4675-acea-247579799b4b, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:49198->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-02-04 13:51:49.215, lastWriteTime=2024-02-04 13:51:48.739, closedTime=2024-02-04 13:51:49.219, connected server version=5.1}
   2024-02-04 13:51:49,222 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
   2024-02-04 13:51:49,227 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
   2024-02-04 13:51:49,228 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
   2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
   
   ===============================================================================
   
   
   2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error,
   
   2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/seatunnel/issues
   
   2024-02-04 13:51:49,228 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed
   
   2024-02-04 13:51:49,230 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   Caused by: java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Compressed class space
           at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
           at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
           at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
           at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
           at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243)
           at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245)
           at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135)
           at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
           at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
           at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
           at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
           at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
           at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
           at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
   Caused by: java.lang.OutOfMemoryError: Compressed class space
   
   2024-02-04 13:51:49,230 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
   ===============================================================================
   
   
   
   Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   Caused by: java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Compressed class space
           at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
           at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
           at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
           at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
           at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
           at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251)
           at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243)
           at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245)
           at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135)
           at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
           at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
           at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
           at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
           at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
           at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
           at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
   Caused by: java.lang.OutOfMemoryError: Compressed class space
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   Zeta
   
   ### Java or Scala Version
   
   jdk1.8
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] [Zeta] OutOfMemoryError: Compressed class space [seatunnel]

Posted by "liugddx (via GitHub)" <gi...@apache.org>.

liugddx commented on issue #6333:
URL: https://github.com/apache/seatunnel/issues/6333#issuecomment-1928858753

   > > Use 2.3.4.
   > > ok, use 2.3.4.
   > 
   > Each time a seatunnel task is run, the loaded class of SeaTunnelServer is incremented, but it is not released after the task is executed.
   > 
   > ![image](https://private-user-images.githubusercontent.com/26395835/302518110-0be8969c-a861-4d2c-8faf-49bf8b76342d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDcyMDA1MTYsIm5iZiI6MTcwNzIwMDIxNiwicGF0aCI6Ii8yNjM5NTgzNS8zMDI1MTgxMTAtMGJlODk2OWMtYTg2MS00ZDJjLThmYWYtNDliZjhiNzYzNDJkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAyMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMjA2VDA2MTY1NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRiNzBkYjdiZmY1ODk1NGFiMWY1YWVjYmQwNGQ0OTBiMzA3ZTZlOGMzNGJhMmQwOWIwYzI3YjkzYWIzYjhmNDMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.61nQ_GQush4R-6AQi4kNDzMzuP7y25fuyOFo6TAkuoU)
   
   This issue has been fixed, please check #6334 5773
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org