You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/09 05:56:56 UTC
[GitHub] [incubator-seatunnel] isMrH opened a new issue, #2698: [Bug] [Module Name] Bug title
isMrH opened a new issue, #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
I encountered an exception when using sink tidb.
Because I can't control the batch size when I use `Spark sink JDBC` to connect to `tidb`, the transaction is too large when I insert tidb, resulting in memory overflow, so I want to use `Sink tidb` for data synchronization of billion level, but an error occurs and I can't continue.
### SeaTunnel Version
2.1.2
### SeaTunnel Config
```conf
env {
spark.app.name = "com"
spark.executor.instances = 10
spark.executor.cores = 4
spark.executor.memory = "3g"
spark.sql.catalogImplementation = "hive"
spark.tispark.pd.addresses = "10.32.48.1:2379,10.32.48.2:2379,10.32.48.3:2379"
spark.sql.extensions = "org.apache.spark.sql.TiExtensions"
}
source {
hive {
pre_sql = """
select id from test limit 10
"""
result_table_name = "test"
}
}
transform {
}
sink {
tidb {
addr = "10.32.48.xx",
port = "4000"
database = "test"
table = "test"
user = "user "
password = "password "
source_table_name = "test"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark.sh -c ./config/st_export_test.conf -e cluster -m yarn
```
### Error Exception
```log
[2022-09-09 13:47:19.807]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
tPromise.tryFailure(Promise.scala:157)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:692)
Caused by: java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3202)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3247)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3286)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1914)
at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:74)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:531)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:106)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:43)
at org.apache.seatunnel.core.base.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:61)
at org.apache.seatunnel.core.base.config.ExecutionContext.<init>(ExecutionContext.java:49)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:62)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
22/09/09 13:47:18 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://cdh-09.qd.link-x.host:8020/user/root/.sparkStaging/application_1662097049317_8027
22/09/09 13:47:19 INFO storage.DiskBlockManager: Shutdown hook called
22/09/09 13:47:19 INFO util.ShutdownHookManager: Shutdown hook called
22/09/09 13:47:19 INFO util.ShutdownHookManager: Deleting directory /data/yarn/nm/usercache/root/appcache/application_1662097049317_8027/spark-f4c41e1d-e33f-41bb-87c7-c1392b96741d
22/09/09 13:47:19 INFO util.ShutdownHookManager: Deleting directory /data/yarn/nm/usercache/root/appcache/application_1662097049317_8027/spark-f4c41e1d-e33f-41bb-87c7-c1392b96741d/userFiles-77997d06-f58d-429f-9c01-72196f3658bf
```
### Flink or Spark Version
Spark version 2.4.0-cdh6.2.1
### Java or Scala Version
Scala version 2.11.12
### Screenshots
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1241603011
@isMrH You can try to manually remove connectors/spark/seatunnel - connector - spark - tidb - 2.1.2. The hadoop jar package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] isMrH commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
isMrH commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1246191180
@Carl-Zhou-CN I don't know if it's related to this issue
Exception when primary key or unique key exists in `tidb` table.
```
sink {
tidb {
addr = "10.32.48.xx",
port = "4000"
database = "test"
table = "test"
user = "user "
password = "password "
source_table_name = "test"
replace = "true"
}
}
# tidb
`id` int(11) NOT NULL AUTO_INCREMENT COMMENT '自增ID',
PRIMARY KEY (`id`),
UNIQUE KEY `uk_ip_id` (`ip_id`)
```
```
22/09/14 11:28:03 INFO yarn.Client:
client token: N/A
diagnostics: User class threw exception: org.apache.seatunnel.core.base.exception.CommandExecuteException: Execute Spark task error
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:70)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: com.pingcap.tikv.exception.GrpcException: retry is exhausted.
at com.pingcap.tikv.util.ConcreteBackOffer.doBackOffWithMaxSleep(ConcreteBackOffer.java:148)
at com.pingcap.tikv.util.ConcreteBackOffer.doBackOff(ConcreteBackOffer.java:119)
at com.pingcap.tikv.allocator.RowIDAllocator.create(RowIDAllocator.java:95)
at com.pingcap.tispark.write.TiBatchWriteTable.getRowIDAllocator(TiBatchWriteTable.scala:395)
at com.pingcap.tispark.write.TiBatchWriteTable.preCalculate(TiBatchWriteTable.scala:201)
at com.pingcap.tispark.write.TiBatchWrite$$anonfun$1.apply(TiBatchWrite.scala:203)
at com.pingcap.tispark.write.TiBatchWrite$$anonfun$1.apply(TiBatchWrite.scala:203)
at scala.collection.immutable.List.map(List.scala:284)
at com.pingcap.tispark.write.TiBatchWrite.doWrite(TiBatchWrite.scala:203)
at com.pingcap.tispark.write.TiBatchWrite.com$pingcap$tispark$write$TiBatchWrite$$write(TiBatchWrite.scala:88)
at com.pingcap.tispark.write.TiBatchWrite$.write(TiBatchWrite.scala:45)
at com.pingcap.tispark.write.TiDBWriter$.write(TiDBWriter.scala:40)
at com.pingcap.tispark.TiDBDataSource.createRelation(TiDBDataSource.scala:57)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
at org.apache.seatunnel.spark.tidb.sink.Tidb.output(Tidb.scala:57)
at org.apache.seatunnel.spark.tidb.sink.Tidb.output(Tidb.scala:30)
at org.apache.seatunnel.spark.SparkEnvironment.sinkProcess(SparkEnvironment.java:179)
at org.apache.seatunnel.spark.batch.SparkBatchExecution.start(SparkBatchExecution.java:54)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:67)
... 7 more
Caused by: java.lang.RuntimeException: java.io.EOFException
at com.pingcap.tikv.codec.CodecDataInput.readLong(CodecDataInput.java:158)
at com.pingcap.tikv.allocator.RowIDAllocator.updateMeta(RowIDAllocator.java:159)
at com.pingcap.tikv.allocator.RowIDAllocator.updateHash(RowIDAllocator.java:196)
at com.pingcap.tikv.allocator.RowIDAllocator.udpateAllocateId(RowIDAllocator.java:230)
at com.pingcap.tikv.allocator.RowIDAllocator.initSigned(RowIDAllocator.java:273)
at com.pingcap.tikv.allocator.RowIDAllocator.doCreate(RowIDAllocator.java:109)
at com.pingcap.tikv.allocator.RowIDAllocator.create(RowIDAllocator.java:90)
... 42 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at com.pingcap.tikv.codec.CodecDataInput.readLong(CodecDataInput.java:156)
... 48 more
ApplicationMaster host: hdfs-05.qd.link-x.host
ApplicationMaster RPC port: 36736
queue: root.users.root
start time: 1663125944895
final status: FAILED
tracking URL: http://cdh-01.qd.link-x.host:8088/proxy/application_1662097049317_11464/
user: root
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] isMrH commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
isMrH commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1241751749
@Carl-Zhou-CN Thank you for your answer.
I removed `connectors/spark/seatunnel-connector-spark-tidb-2.1.2`.
Do I need to do anything else?
I also tried replacing `seatunnel-connector-spark-tidb-2.1.2` with `tispark-assembly-2.4.3-scala_2.11.jar` but it still doesn't work
The following error occurred.
```
22/09/09 17:24:13 INFO yarn.Client:
client token: N/A
diagnostics: User class threw exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Plugin class not found by name :[tidb]
at org.apache.seatunnel.core.base.config.PluginFactory.lambda$createPlugins$7(PluginFactory.java:185)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.apache.seatunnel.core.base.config.PluginFactory.createPlugins(PluginFactory.java:179)
at org.apache.seatunnel.core.base.config.ExecutionContext.<init>(ExecutionContext.java:55)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:62)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: java.lang.ClassNotFoundException: Plugin class not found by name :[tidb]
at org.apache.seatunnel.core.base.config.PluginFactory.createPluginInstanceIgnoreCase(PluginFactory.java:221)
at org.apache.seatunnel.core.base.config.PluginFactory.lambda$createPlugins$7(PluginFactory.java:181)
... 11 more
ApplicationMaster host: cdh-07.qd.link-x.host
ApplicationMaster RPC port: 43493
queue: root.users.root
start time: 1662715437408
final status: FAILED
tracking URL: http://cdh-01.qd.link-x.host:8088/proxy/application_1662097049317_8037/
user: root
22/09/09 17:24:13 ERROR yarn.Client: Application diagnostics message: User class threw exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Plugin class not found by name :[tidb]
at org.apache.seatunnel.core.base.config.PluginFactory.lambda$createPlugins$7(PluginFactory.java:185)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.apache.seatunnel.core.base.config.PluginFactory.createPlugins(PluginFactory.java:179)
at org.apache.seatunnel.core.base.config.ExecutionContext.<init>(ExecutionContext.java:55)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:62)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: java.lang.ClassNotFoundException: Plugin class not found by name :[tidb]
at org.apache.seatunnel.core.base.config.PluginFactory.createPluginInstanceIgnoreCase(PluginFactory.java:221)
at org.apache.seatunnel.core.base.config.PluginFactory.lambda$createPlugins$7(PluginFactory.java:181)
... 11 more
Exception in thread "main" org.apache.spark.SparkException: Application application_1662097049317_8037 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1155)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1603)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/09/09 17:24:13 INFO util.ShutdownHookManager: Shutdown hook called
22/09/09 17:24:13 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-126a149f-8c22-43e0-b80e-5dbfab8a199a
22/09/09 17:24:13 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-ab7f566d-bd3b-4031-9816-5ca2c4a49285
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1241597344
I want to do some screening work for this, can you assign this task to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] isMrH closed issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
isMrH closed issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
URL: https://github.com/apache/incubator-seatunnel/issues/2698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] isMrH commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
isMrH commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1243584707
@Carl-Zhou-CN
Follow the above to the end.
I've now got a new bug.
```
22/09/12 19:01:19 INFO yarn.Client:
client token: N/A
diagnostics: User class threw exception: java.lang.RuntimeException: Execute Spark task error
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 4 times, most recent failure: Lost task 0.3 in stage 8.0 (TID 1742, cdh-08.qd.link-x.host, executor 6): java.io.InvalidClassException: com.pingcap.tispark.write.TiBatchWriteTable; local class incompatible: stream classdesc serialVersionUID = 150732324775698016, local class serialVersionUID = -1234081212328682880
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1241758860
@isMrH Sorry, I may not express it very clearly, it is to delete the related jars of hadoop in seatunnel-connector-spark-tidb-2.1.2, not to remove the entire package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1241768097
jar vtf connectors/spark/seatunnel-connector-spark-tidb-2.1.2.jar | grep hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] isMrH commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
isMrH commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1244797152
@Carl-Zhou-CN Thank you, according to your answer, the problem is solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1241780143
Of course, this is only a temporary solution. I hope this command will be helpful for you to delete it. I will fix this bug in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1244763246
@isMrH hi,isMrH, I see that you closed issues, I want to ask, is this issue finally resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Carl-Zhou-CN commented on issue #2698: [Bug] [Spark Sink tidb] Sink tidb exception
Posted by GitBox <gi...@apache.org>.
Carl-Zhou-CN commented on issue #2698:
URL: https://github.com/apache/incubator-seatunnel/issues/2698#issuecomment-1246276033
@isMrH options.replace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org