You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "18210308633 (via GitHub)" <gi...@apache.org> on 2024/07/23 09:12:44 UTC

[I] 把influxdb里面的数据同步到doris [seatunnel]

18210308633 opened a new issue, #7250:
URL: https://github.com/apache/seatunnel/issues/7250

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   2024-07-23 16:32:32,247 WARN  org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs$MasterTypeValidator - 
   ******************************************************************************************
   -e and --deploy-mode deprecated in 2.3.1, please use -m and --master instead of it
   ******************************************************************************************
   Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
   Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml
   Jul 23, 2024 4:32:32 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
   INFO: seatunnel.home is /home/data/apache-seatunnel-2.3.3
   Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
   Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml
   2024-07-23 16:32:32,606 WARN  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
   2024-07-23 16:32:32,606 INFO  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] Resolving domain name '19.0.037' to address(es): [19.0.0.37]
   2024-07-23 16:32:32,607 INFO  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [19.0.037/19.0.0.37]
   2024-07-23 16:32:32,628 INFO  org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
   2024-07-23 16:32:32,629 INFO  com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
   2024-07-23 16:32:32,630 INFO  com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Cluster name: seatunnel-735458
   2024-07-23 16:32:32,630 INFO  com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] 
                                                            
    _____               _____                             _ 
   /  ___|             |_   _|                           | |
   \ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
    `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
   /\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
   \____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|
                                                            
   
   2024-07-23 16:32:32,630 INFO  com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
   2024-07-23 16:32:32,630 INFO  com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
   To enable integrity checker do one of the following: 
     - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
     - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
     - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
     - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
   2024-07-23 16:32:32,632 INFO  com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] The Jet engine is disabled.
   To enable the Jet engine on the members, do one of the following:
     - Change member config using Java API: config.getJetConfig().setEnabled(true)
     - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
     - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
     - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
   2024-07-23 16:32:32,949 INFO  com.hazelcast.system.security - [19.0.037]:5801 [seatunnel-735458] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
   2024-07-23 16:32:33,002 INFO  com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Using TCP/IP discovery
   2024-07-23 16:32:33,003 WARN  com.hazelcast.cp.CPSubsystem - [19.0.037]:5801 [seatunnel-735458] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
   Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
   Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml
   Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
   Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml
   2024-07-23 16:32:33,159 WARN  org.apache.seatunnel.engine.server.TaskExecutionService - [19.0.037]:5801 [seatunnel-735458] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
   2024-07-23 16:32:33,159 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [19.0.037]:5801 [seatunnel-735458] [5.1] Created new BusWork : 1083086553
   2024-07-23 16:32:33,173 WARN  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [19.0.037]:5801
   2024-07-23 16:32:33,176 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [19.0.037]:5801 [seatunnel-735458] [5.1] 
   ***********************************************
        CoordinatorService Thread Pool Status
   ***********************************************
   activeCount               :                   0
   corePoolSize              :                   0
   maximumPoolSize           :          2147483647
   poolSize                  :                   0
   completedTaskCount        :                   0
   taskCount                 :                   0
   ***********************************************
   
   2024-07-23 16:32:33,211 INFO  com.hazelcast.internal.diagnostics.Diagnostics - [19.0.037]:5801 [seatunnel-735458] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
   2024-07-23 16:32:33,217 INFO  com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is STARTING
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5816 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5814 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5812 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5810 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5822 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5820 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5824 is added to the blacklist.
   2024-07-23 16:32:33,244 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5818 is added to the blacklist.
   2024-07-23 16:32:33,245 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5830 is added to the blacklist.
   2024-07-23 16:32:33,245 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5828 is added to the blacklist.
   2024-07-23 16:32:33,246 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5826 is added to the blacklist.
   2024-07-23 16:32:33,246 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5807 is added to the blacklist.
   2024-07-23 16:32:33,246 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5805 is added to the blacklist.
   2024-07-23 16:32:33,246 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5803 is added to the blacklist.
   2024-07-23 16:32:33,246 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5815 is added to the blacklist.
   2024-07-23 16:32:33,246 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5813 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5811 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5823 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5821 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5817 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5819 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5827 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5829 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5825 is added to the blacklist.
   2024-07-23 16:32:33,248 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5806 is added to the blacklist.
   2024-07-23 16:32:33,248 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5804 is added to the blacklist.
   2024-07-23 16:32:33,247 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5809 is added to the blacklist.
   2024-07-23 16:32:33,248 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5802 is added to the blacklist.
   2024-07-23 16:32:33,248 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5808 is added to the blacklist.
   2024-07-23 16:32:33,255 WARN  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35139, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
   2024-07-23 16:32:33,257 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35139, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
   2024-07-23 16:32:33,626 WARN  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=2, /127.0.0.1:5801->/127.0.0.1:60975, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
   2024-07-23 16:32:33,627 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=2, /127.0.0.1:5801->/127.0.0.1:60975, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
   2024-07-23 16:32:34,248 INFO  com.hazelcast.internal.cluster.ClusterService - [19.0.037]:5801 [seatunnel-735458] [5.1] 
   
   Members {size:1, ver:1} [
           Member [19.0.037]:5801 - feb828c8-a852-47eb-8c0b-68b497c2e6e1 this
   ]
   
   2024-07-23 16:32:34,258 INFO  com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is STARTED
   Jul 23, 2024 4:32:34 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
   Jul 23, 2024 4:32:34 PM com.hazelcast.internal.config.AbstractConfigLocator
   INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
   2024-07-23 16:32:34,330 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-735458] [5.1] Running with 2 response threads, dynamic=true
   2024-07-23 16:32:34,340 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
   2024-07-23 16:32:34,341 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
   2024-07-23 16:32:34,345 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Trying to connect to cluster: seatunnel-735458
   2024-07-23 16:32:34,346 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Trying to connect to [19.0.0.37]:5801
   2024-07-23 16:32:34,350 WARN  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=4, /127.0.0.1:5801->/127.0.0.1:44547, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
   2024-07-23 16:32:34,351 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=4, /127.0.0.1:5801->/127.0.0.1:44547, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
   2024-07-23 16:32:34,351 INFO  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.37]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: d01daf7d-5edf-4225-b98c-630dc83d9ba3, client name: hz.client_1, client version: 5.1
   2024-07-23 16:32:34,353 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
   2024-07-23 16:32:34,353 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Authenticated with server [19.0.037]:5801:feb828c8-a852-47eb-8c0b-68b497c2e6e1, server version: 5.1, local address: /19.0.0.37:45643
   2024-07-23 16:32:34,354 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-735458] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
   2024-07-23 16:32:34,359 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-735458] [5.1] 
   
   Members [1] {
           Member [19.0.037]:5801 - feb828c8-a852-47eb-8c0b-68b497c2e6e1
   }
   
   2024-07-23 16:32:34,379 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
   2024-07-23 16:32:34,483 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
   2024-07-23 16:32:34,498 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/influxdb_to_doris.conf
   2024-07-23 16:32:34,534 INFO  org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
   2024-07-23 16:32:34,569 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
       "env" : {
           "parallelism" : 1,
           "job.mode" : "BATCH",
           "checkpoint.interval" : 10000
       },
       "source" : [
           {
               "schema" : {
                   "fields" : {
                       "node_cpu_seconds_total" : "STRING",
                       "pg_database_size_bytes" : "STRING",
                       "node_memory_MemTotal_bytes" : "STRING"
                   }
               },
               "database" : "sbyw",
               "password" : "influx123",
               "lower_bound" : 1,
               "source_table_name" : [
                   "prometheus_remote_write"
               ],
               "result_table_name" : [
                   "sbyw_application_process_type"
               ],
               "upper_bound" : 100,
               "plugin_name" : "InfluxDB",
               "user" : "influx",
               "url" : "http://19.0.0.36:8086",
               "sql" : "SELECT * FROM prometheus_remote_write WHERE time > now() - 1h",
               "partition_num" : 10
           }
       ],
       "sink" : [
           {
               "password" : "sbyw@123456",
               "fenodes" : [
                   "19.0.0.37:8030"
               ],
               "sink.enable-2pc" : "true",
               "doris.config" : {
                   "format" : "json",
                   "read_json_by_line" : "true"
               },
               "table.identifier" : "sbyw_data_acquisition.sbyw_application_process_type",
               "source_table_name" : [
                   "prometheus_remote_write"
               ],
               "sink.enable-delete" : "true",
               "plugin_name" : "Doris",
               "sink.label-prefix" : "test-cdc",
               "username" : "root"
           }
       ],
       "transform" : [
           {
               "source_table_name" : [
                   "prometheus_remote_write"
               ],
               "result_table_name" : [
                   "sbyw_application_process_type"
               ],
               "fields" : {
                   "system_cpu_usage" : "node_cpu_seconds_total",
                   "process_occupy_physical_memory_size" : "pg_database_size_bytes",
                   "process_read_written_file_system_total_bytes" : "node_memory_MemTotal_bytes"
               },
               "plugin_name" : "Copy"
           }
       ]
   }
   
   2024-07-23 16:32:34,588 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-07-23 16:32:34,589 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-07-23 16:32:34,592 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /home/data/apache-seatunnel-2.3.5/connectors/seatunnel
   2024-07-23 16:32:34,596 INFO  org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
   2024-07-23 16:32:34,596 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-07-23 16:32:34,597 ERROR org.apache.seatunnel.engine.core.parse.ConfigParserUtil - The 'source_table_name' option configured in [Copy] is incorrect, and the source/transform[prometheus_remote_write] is not found.
   2024-07-23 16:32:34,597 INFO  org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
   2024-07-23 16:32:34,597 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-07-23 16:32:34,597 ERROR org.apache.seatunnel.engine.core.parse.ConfigParserUtil - The 'source_table_name' option configured in [Doris] is incorrect, and the source/transform[prometheus_remote_write] is not found.
   2024-07-23 16:32:34,597 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
   2024-07-23 16:32:34,598 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
   2024-07-23 16:32:34,626 WARN  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=5, /127.0.0.1:5801->/127.0.0.1:39627, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
   2024-07-23 16:32:34,627 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=5, /127.0.0.1:5801->/127.0.0.1:39627, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
   2024-07-23 16:32:34,635 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /home/data/apache-seatunnel-2.3.5/connectors/seatunnel
   2024-07-23 16:32:34,675 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath
   2024-07-23 16:32:34,921 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
   2024-07-23 16:32:34,923 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.37]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
   2024-07-23 16:32:34,923 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Removed connection to endpoint: [19.0.037]:5801:feb828c8-a852-47eb-8c0b-68b497c2e6e1, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/19.0.0.37:45643->/19.0.0.37:5801}, remoteAddress=[19.0.037]:5801, lastReadTime=2024-07-23 16:32:34.493, lastWriteTime=2024-07-23 16:32:34.492, closedTime=2024-07-23 16:32:34.921, connected server version=5.1}
   2024-07-23 16:32:34,923 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
   2024-07-23 16:32:34,924 INFO  com.hazelcast.client.impl.ClientEndpointManager - [19.0.037]:5801 [seatunnel-735458] [5.1] Destroying ClientEndpoint{connection=Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.37]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1721723554350, latest clientAttributes=lastStatisticsCollectionTime=1721723554379,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1721723554347,clientAddress=19.0.0.37,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=11341529088,os.freePhysicalMemorySize=45426728960,os.freeSwapSpaceSize=8535400448,os.maxFileDescriptorCount=65535,os.openFileDescriptorCount=122,os.processCpuTime=5410000000,os.systemLoadAverage=0.16,os.totalPhysicalMemorySize=68326711296,os
 .totalSwapSpaceSize=8535400448,runtime.availableProcessors=16,runtime.freeMemory=968396248,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=2607,runtime.usedMemory=60781096, labels=[]}
   2024-07-23 16:32:34,925 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
   2024-07-23 16:32:34,925 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
   2024-07-23 16:32:34,925 INFO  com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTTING_DOWN
   2024-07-23 16:32:34,928 INFO  com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Shutting down connection manager...
   2024-07-23 16:32:34,929 INFO  com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Shutting down node engine...
   2024-07-23 16:32:34,933 INFO  org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted
   2024-07-23 16:32:37,944 INFO  com.hazelcast.instance.impl.NodeExtension - [19.0.037]:5801 [seatunnel-735458] [5.1] Destroying node NodeExtension.2024-07-23 16:32:37,945 INFO  com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Hazelcast Shutdown is completed in 3017 ms.
   2024-07-23 16:32:37,945 INFO  com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTDOWN
   2024-07-23 16:32:37,945 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
   Exception in thread "main" java.lang.NoSuchFieldError: Companion
           at okhttp3.internal.Util.<clinit>(Util.kt:71)
           at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1239)
           at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1634)
           at okhttp3.HttpUrl.get(HttpUrl.kt)
           at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554)
           at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:178)
           at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:120)
           at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:72)
           at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:89)
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
           at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   
   ### SeaTunnel Version
   
   SeaTunnel 2.3.3
   
   ### SeaTunnel Config
   
   ```conf
   # influxdb_to_doris.conf
   
   env {
     parallelism = 1
     job.mode = "BATCH"
     checkpoint.interval = 10000
   }
   
   source {
   
       InfluxDB {
           url = "http://x.x.x.x:8086"
           sql = "SELECT node_cpu_seconds_total,pg_database_size_bytes,node_memory_MemTotal_bytes FROM prometheus_remote_write WHERE time > now() - 1h"
           database = "sbyw"
   	user = "influx"
           password = "influx123"
           upper_bound = 100
           lower_bound = 1
           partition_num = 10
           source_table_name = ["prometheus_remote_write"]
           result_table_name = ["sbyw_application_process_type"]
           #split_column = "value"
           schema {
               fields {
   		node_cpu_seconds_total = STRING
                   pg_database_size_bytes = STRING
                   node_memory_MemTotal_bytes = STRING
               }
            }
   	
       }
   
   }
   
   sink {
     Doris {
       fenodes = ["x.x.x.x:8030"]
       username = "root"
       password = "sbyw@123456"
       table.identifier = "sbyw_data_acquisition.sbyw_application_process_type"
       sink.label-prefix = "test-cdc"
       sink.enable-2pc = "true"
       sink.enable-delete = "true"
       source_table_name = ["prometheus_remote_write"]
       doris.config {
         format = "json"
         read_json_by_line = "true"
       }
     }
   }
   
   transform {
     Copy {
       source_table_name = ["prometheus_remote_write"]
       result_table_name = ["sbyw_application_process_type"]
       fields {
         system_cpu_usage = node_cpu_seconds_total
         process_occupy_physical_memory_size = pg_database_size_bytes
         process_read_written_file_system_total_bytes = node_memory_MemTotal_bytes
       }
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh -f --config ./config/influxdb_to_doris.conf -e local
   ```
   
   
   ### Error Exception
   
   ```log
   2024-07-23 16:32:37,945 INFO  com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTDOWN
   2024-07-23 16:32:37,945 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
   Exception in thread "main" java.lang.NoSuchFieldError: Companion
           at okhttp3.internal.Util.<clinit>(Util.kt:71)
           at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1239)
           at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1634)
           at okhttp3.HttpUrl.get(HttpUrl.kt)
           at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554)
           at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:178)
           at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:120)
           at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:72)
           at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:89)
           at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
           at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
           at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
           at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
           at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
           at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
           at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 把influxdb里面的数据同步到doris [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7250: 把influxdb里面的数据同步到doris
URL: https://github.com/apache/seatunnel/issues/7250


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 把influxdb里面的数据同步到doris [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7250:
URL: https://github.com/apache/seatunnel/issues/7250#issuecomment-2305960025

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 把influxdb里面的数据同步到doris [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7250:
URL: https://github.com/apache/seatunnel/issues/7250#issuecomment-2319540742

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org