You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "18210308633 (via GitHub)" <gi...@apache.org> on 2024/07/23 09:12:44 UTC
[I] 把influxdb里面的数据同步到doris [seatunnel]
18210308633 opened a new issue, #7250:
URL: https://github.com/apache/seatunnel/issues/7250
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
2024-07-23 16:32:32,247 WARN org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs$MasterTypeValidator -
******************************************************************************************
-e and --deploy-mode deprecated in 2.3.1, please use -m and --master instead of it
******************************************************************************************
Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml
Jul 23, 2024 4:32:32 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
INFO: seatunnel.home is /home/data/apache-seatunnel-2.3.3
Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml
2024-07-23 16:32:32,606 WARN com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2024-07-23 16:32:32,606 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] Resolving domain name '19.0.037' to address(es): [19.0.0.37]
2024-07-23 16:32:32,607 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [19.0.037/19.0.0.37]
2024-07-23 16:32:32,628 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
2024-07-23 16:32:32,629 INFO com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Cluster name: seatunnel-735458
2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1]
_____ _____ _
/ ___| |_ _| | |
\ `--. ___ __ _ | | _ _ _ __ _ __ ___ | |
`--. \ / _ \ / _` | | | | | | || '_ \ | '_ \ / _ \| |
/\__/ /| __/| (_| | | | | |_| || | | || | | || __/| |
\____/ \___| \__,_| \_/ \__,_||_| |_||_| |_| \___||_|
2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-07-23 16:32:32,632 INFO com.hazelcast.system - [19.0.037]:5801 [seatunnel-735458] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-07-23 16:32:32,949 INFO com.hazelcast.system.security - [19.0.037]:5801 [seatunnel-735458] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2024-07-23 16:32:33,002 INFO com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Using TCP/IP discovery
2024-07-23 16:32:33,003 WARN com.hazelcast.cp.CPSubsystem - [19.0.037]:5801 [seatunnel-735458] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml
Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml
2024-07-23 16:32:33,159 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [19.0.037]:5801 [seatunnel-735458] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2024-07-23 16:32:33,159 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [19.0.037]:5801 [seatunnel-735458] [5.1] Created new BusWork : 1083086553
2024-07-23 16:32:33,173 WARN org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [19.0.037]:5801
2024-07-23 16:32:33,176 INFO org.apache.seatunnel.engine.server.CoordinatorService - [19.0.037]:5801 [seatunnel-735458] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 0
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 0
completedTaskCount : 0
taskCount : 0
***********************************************
2024-07-23 16:32:33,211 INFO com.hazelcast.internal.diagnostics.Diagnostics - [19.0.037]:5801 [seatunnel-735458] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-07-23 16:32:33,217 INFO com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is STARTING
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5816 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5814 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5812 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5810 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5822 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5820 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5824 is added to the blacklist.
2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5818 is added to the blacklist.
2024-07-23 16:32:33,245 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5830 is added to the blacklist.
2024-07-23 16:32:33,245 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5828 is added to the blacklist.
2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5826 is added to the blacklist.
2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5807 is added to the blacklist.
2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5805 is added to the blacklist.
2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5803 is added to the blacklist.
2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5815 is added to the blacklist.
2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5813 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5811 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5823 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5821 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5817 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5819 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5827 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5829 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5825 is added to the blacklist.
2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5806 is added to the blacklist.
2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5804 is added to the blacklist.
2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5809 is added to the blacklist.
2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5802 is added to the blacklist.
2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5808 is added to the blacklist.
2024-07-23 16:32:33,255 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35139, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
2024-07-23 16:32:33,257 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35139, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-07-23 16:32:33,626 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=2, /127.0.0.1:5801->/127.0.0.1:60975, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
2024-07-23 16:32:33,627 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=2, /127.0.0.1:5801->/127.0.0.1:60975, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-07-23 16:32:34,248 INFO com.hazelcast.internal.cluster.ClusterService - [19.0.037]:5801 [seatunnel-735458] [5.1]
Members {size:1, ver:1} [
Member [19.0.037]:5801 - feb828c8-a852-47eb-8c0b-68b497c2e6e1 this
]
2024-07-23 16:32:34,258 INFO com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is STARTED
Jul 23, 2024 4:32:34 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
Jul 23, 2024 4:32:34 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
2024-07-23 16:32:34,330 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-735458] [5.1] Running with 2 response threads, dynamic=true
2024-07-23 16:32:34,340 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2024-07-23 16:32:34,341 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2024-07-23 16:32:34,345 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Trying to connect to cluster: seatunnel-735458
2024-07-23 16:32:34,346 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Trying to connect to [19.0.0.37]:5801
2024-07-23 16:32:34,350 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=4, /127.0.0.1:5801->/127.0.0.1:44547, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
2024-07-23 16:32:34,351 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=4, /127.0.0.1:5801->/127.0.0.1:44547, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-07-23 16:32:34,351 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.37]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: d01daf7d-5edf-4225-b98c-630dc83d9ba3, client name: hz.client_1, client version: 5.1
2024-07-23 16:32:34,353 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2024-07-23 16:32:34,353 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Authenticated with server [19.0.037]:5801:feb828c8-a852-47eb-8c0b-68b497c2e6e1, server version: 5.1, local address: /19.0.0.37:45643
2024-07-23 16:32:34,354 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-735458] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-07-23 16:32:34,359 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-735458] [5.1]
Members [1] {
Member [19.0.037]:5801 - feb828c8-a852-47eb-8c0b-68b497c2e6e1
}
2024-07-23 16:32:34,379 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2024-07-23 16:32:34,483 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
2024-07-23 16:32:34,498 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/influxdb_to_doris.conf
2024-07-23 16:32:34,534 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2024-07-23 16:32:34,569 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"parallelism" : 1,
"job.mode" : "BATCH",
"checkpoint.interval" : 10000
},
"source" : [
{
"schema" : {
"fields" : {
"node_cpu_seconds_total" : "STRING",
"pg_database_size_bytes" : "STRING",
"node_memory_MemTotal_bytes" : "STRING"
}
},
"database" : "sbyw",
"password" : "influx123",
"lower_bound" : 1,
"source_table_name" : [
"prometheus_remote_write"
],
"result_table_name" : [
"sbyw_application_process_type"
],
"upper_bound" : 100,
"plugin_name" : "InfluxDB",
"user" : "influx",
"url" : "http://19.0.0.36:8086",
"sql" : "SELECT * FROM prometheus_remote_write WHERE time > now() - 1h",
"partition_num" : 10
}
],
"sink" : [
{
"password" : "sbyw@123456",
"fenodes" : [
"19.0.0.37:8030"
],
"sink.enable-2pc" : "true",
"doris.config" : {
"format" : "json",
"read_json_by_line" : "true"
},
"table.identifier" : "sbyw_data_acquisition.sbyw_application_process_type",
"source_table_name" : [
"prometheus_remote_write"
],
"sink.enable-delete" : "true",
"plugin_name" : "Doris",
"sink.label-prefix" : "test-cdc",
"username" : "root"
}
],
"transform" : [
{
"source_table_name" : [
"prometheus_remote_write"
],
"result_table_name" : [
"sbyw_application_process_type"
],
"fields" : {
"system_cpu_usage" : "node_cpu_seconds_total",
"process_occupy_physical_memory_size" : "pg_database_size_bytes",
"process_read_written_file_system_total_bytes" : "node_memory_MemTotal_bytes"
},
"plugin_name" : "Copy"
}
]
}
2024-07-23 16:32:34,588 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2024-07-23 16:32:34,589 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2024-07-23 16:32:34,592 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /home/data/apache-seatunnel-2.3.5/connectors/seatunnel
2024-07-23 16:32:34,596 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2024-07-23 16:32:34,596 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2024-07-23 16:32:34,597 ERROR org.apache.seatunnel.engine.core.parse.ConfigParserUtil - The 'source_table_name' option configured in [Copy] is incorrect, and the source/transform[prometheus_remote_write] is not found.
2024-07-23 16:32:34,597 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2024-07-23 16:32:34,597 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2024-07-23 16:32:34,597 ERROR org.apache.seatunnel.engine.core.parse.ConfigParserUtil - The 'source_table_name' option configured in [Doris] is incorrect, and the source/transform[prometheus_remote_write] is not found.
2024-07-23 16:32:34,597 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
2024-07-23 16:32:34,598 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2024-07-23 16:32:34,626 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.037]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=5, /127.0.0.1:5801->/127.0.0.1:39627, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed
2024-07-23 16:32:34,627 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=5, /127.0.0.1:5801->/127.0.0.1:39627, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-07-23 16:32:34,635 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /home/data/apache-seatunnel-2.3.5/connectors/seatunnel
2024-07-23 16:32:34,675 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath
2024-07-23 16:32:34,921 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-07-23 16:32:34,923 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.037]:5801 [seatunnel-735458] [5.1] Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.37]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-07-23 16:32:34,923 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Removed connection to endpoint: [19.0.037]:5801:feb828c8-a852-47eb-8c0b-68b497c2e6e1, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/19.0.0.37:45643->/19.0.0.37:5801}, remoteAddress=[19.0.037]:5801, lastReadTime=2024-07-23 16:32:34.493, lastWriteTime=2024-07-23 16:32:34.492, closedTime=2024-07-23 16:32:34.921, connected server version=5.1}
2024-07-23 16:32:34,923 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2024-07-23 16:32:34,924 INFO com.hazelcast.client.impl.ClientEndpointManager - [19.0.037]:5801 [seatunnel-735458] [5.1] Destroying ClientEndpoint{connection=Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.37]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1721723554350, latest clientAttributes=lastStatisticsCollectionTime=1721723554379,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1721723554347,clientAddress=19.0.0.37,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=11341529088,os.freePhysicalMemorySize=45426728960,os.freeSwapSpaceSize=8535400448,os.maxFileDescriptorCount=65535,os.openFileDescriptorCount=122,os.processCpuTime=5410000000,os.systemLoadAverage=0.16,os.totalPhysicalMemorySize=68326711296,os
.totalSwapSpaceSize=8535400448,runtime.availableProcessors=16,runtime.freeMemory=968396248,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=2607,runtime.usedMemory=60781096, labels=[]}
2024-07-23 16:32:34,925 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2024-07-23 16:32:34,925 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2024-07-23 16:32:34,925 INFO com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTTING_DOWN
2024-07-23 16:32:34,928 INFO com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Shutting down connection manager...
2024-07-23 16:32:34,929 INFO com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Shutting down node engine...
2024-07-23 16:32:34,933 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted
2024-07-23 16:32:37,944 INFO com.hazelcast.instance.impl.NodeExtension - [19.0.037]:5801 [seatunnel-735458] [5.1] Destroying node NodeExtension.2024-07-23 16:32:37,945 INFO com.hazelcast.instance.impl.Node - [19.0.037]:5801 [seatunnel-735458] [5.1] Hazelcast Shutdown is completed in 3017 ms.
2024-07-23 16:32:37,945 INFO com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTDOWN
2024-07-23 16:32:37,945 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
Exception in thread "main" java.lang.NoSuchFieldError: Companion
at okhttp3.internal.Util.<clinit>(Util.kt:71)
at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1239)
at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1634)
at okhttp3.HttpUrl.get(HttpUrl.kt)
at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554)
at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:178)
at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:120)
at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:72)
at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:89)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
### SeaTunnel Version
SeaTunnel 2.3.3
### SeaTunnel Config
```conf
# influxdb_to_doris.conf
env {
parallelism = 1
job.mode = "BATCH"
checkpoint.interval = 10000
}
source {
InfluxDB {
url = "http://x.x.x.x:8086"
sql = "SELECT node_cpu_seconds_total,pg_database_size_bytes,node_memory_MemTotal_bytes FROM prometheus_remote_write WHERE time > now() - 1h"
database = "sbyw"
user = "influx"
password = "influx123"
upper_bound = 100
lower_bound = 1
partition_num = 10
source_table_name = ["prometheus_remote_write"]
result_table_name = ["sbyw_application_process_type"]
#split_column = "value"
schema {
fields {
node_cpu_seconds_total = STRING
pg_database_size_bytes = STRING
node_memory_MemTotal_bytes = STRING
}
}
}
}
sink {
Doris {
fenodes = ["x.x.x.x:8030"]
username = "root"
password = "sbyw@123456"
table.identifier = "sbyw_data_acquisition.sbyw_application_process_type"
sink.label-prefix = "test-cdc"
sink.enable-2pc = "true"
sink.enable-delete = "true"
source_table_name = ["prometheus_remote_write"]
doris.config {
format = "json"
read_json_by_line = "true"
}
}
}
transform {
Copy {
source_table_name = ["prometheus_remote_write"]
result_table_name = ["sbyw_application_process_type"]
fields {
system_cpu_usage = node_cpu_seconds_total
process_occupy_physical_memory_size = pg_database_size_bytes
process_read_written_file_system_total_bytes = node_memory_MemTotal_bytes
}
}
}
```
### Running Command
```shell
./bin/seatunnel.sh -f --config ./config/influxdb_to_doris.conf -e local
```
### Error Exception
```log
2024-07-23 16:32:37,945 INFO com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTDOWN
2024-07-23 16:32:37,945 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
Exception in thread "main" java.lang.NoSuchFieldError: Companion
at okhttp3.internal.Util.<clinit>(Util.kt:71)
at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1239)
at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1634)
at okhttp3.HttpUrl.get(HttpUrl.kt)
at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554)
at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:178)
at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:120)
at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:72)
at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:89)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 把influxdb里面的数据同步到doris [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7250: 把influxdb里面的数据同步到doris
URL: https://github.com/apache/seatunnel/issues/7250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 把influxdb里面的数据同步到doris [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7250:
URL: https://github.com/apache/seatunnel/issues/7250#issuecomment-2305960025
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 把influxdb里面的数据同步到doris [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7250:
URL: https://github.com/apache/seatunnel/issues/7250#issuecomment-2319540742
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org