You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "alextinng (via GitHub)" <gi...@apache.org> on 2023/12/13 02:58:36 UTC
[I] [Improvement][tdengine] connector-tdengine didn't work cause no suitable driver found exception and NPE [seatunnel]
alextinng opened a new issue, #5998:
URL: https://github.com/apache/seatunnel/issues/5998
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
unfortunately I got problem connect to tdengine, I tried to find out the cause, but the reason is not printed.
part of log:
```
2023-12-12 19:57:29,452 ERROR org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (786924607960514561), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-TDengine-src]-SourceTask (1/1)] end with state FAILED and Exception: org.apache.seatunnel.connectors.seatunnel.tdengine.exception.TDengineConnectorException: ErrorCode:[COMMON-12], ErrorDescription:[Source reader operation failed, such as (open, close) etc...] - get TDengine connection failed:jdbc:TAOS-RS://localhost:6041/test?user=root&password=taosdata
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.open(TDengineSourceReader.java:113)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.open(SourceFlowLifeCycle.java:115)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:146)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:89)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2023-12-12 19:57:29,452 ERROR org.apache.seatunnel.engine.server.dag.physical.SubPlan - Task TaskGroupLocation{jobId=786924607960514561, pipelineId=1, taskGroupId=50000} Failed in Job SeaTunnel_Job (786924607960514561), Pipeline: [(1/1)], Begin to cancel other tasks in this pipeline.
```
Next I changed the try catch block, add cause before re-throwing SQLException
I re-run the job and this time cause is printed on the screen
```
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:188)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.tdengine.exception.TDengineConnectorException: ErrorCode:[COMMON-12], ErrorDescription:[Source reader operation failed, such as (open, close) etc...] - get TDengine connection failed:jdbc:TAOS-RS://10.5.2.33:30615/vpp?user=root&password=taosdata
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.open(TDengineSourceReader.java:102)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.open(SourceFlowLifeCycle.java:115)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:146)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:89)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:TAOS-RS://10.5.2.33:30615/vpp?user=root&password=taosdata
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.open(TDengineSourceReader.java:100)
... 9 more
at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:122)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
... 2 more
```
this seems to be a simple problem, so I changed the source code again:
......
run the job and get another exception:
```
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:188)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.tdengine.exception.TDengineConnectorException: ErrorCode:[COMMON-12], ErrorDescription:[Source reader operation failed, such as (open, close) etc...] - TDengine split read error
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.lambda$pollNext$0(TDengineSourceReader.java:66)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.pollNext(TDengineSourceReader.java:61)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:135)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:84)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:165)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:89)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.convertDataType(TDengineSourceReader.java:145)
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.read(TDengineSourceReader.java:136)
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.lambda$pollNext$0(TDengineSourceReader.java:64)
... 12 more
at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:122)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
... 2 more
```
......
after debug and found the cause is row columns existing null values, convertDataType will throw NPE if parameter is null
......
coding -> run the job -> and this time I succeed!
### SeaTunnel Version
2.3.2
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
job.mode = "BATCH"
}
source {
TDengine {
url : "jdbc:TAOS-RS://localhost:6041/"
username : "root"
password : "taosdata"
database : "test"
stable : "xxxxx"
lower_bound : "2023-12-12 14:38:05.000"
upper_bound : "2023-12-12 14:38:16.800"
result_table_name = "src"
}
}
sink {
Console {}
}
```
### Running Command
```shell
bin/seatunnel.sh --config a.conf -e local
```
### Error Exception
```log
2023-12-12 19:57:29,452 ERROR org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (786924607960514561), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-TDengine-src]-SourceTask (1/1)] end with state FAILED and Exception: org.apache.seatunnel.connectors.seatunnel.tdengine.exception.TDengineConnectorException: ErrorCode:[COMMON-12], ErrorDescription:[Source reader operation failed, such as (open, close) etc...] - get TDengine connection failed:jdbc:TAOS-RS://localhost:6041/test?user=root&password=taosdata
at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSourceReader.open(TDengineSourceReader.java:113)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.open(SourceFlowLifeCycle.java:115)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:146)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:89)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:613)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [bug][tdengine] connector-tdengine didn't work cause no suitable driver found exception and NPE [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5998:
URL: https://github.com/apache/seatunnel/issues/5998#issuecomment-1907139660
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [bug][tdengine] connector-tdengine didn't work cause no suitable driver found exception and NPE [seatunnel]
Posted by "kulame (via GitHub)" <gi...@apache.org>.
kulame commented on issue #5998:
URL: https://github.com/apache/seatunnel/issues/5998#issuecomment-1913516719
这个问题解决了吗？ 我也遇到了类似的错误。
```
2024-01-28 16:35:02,200 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - report error from task
org.apache.seatunnel.common.utils.SeaTunnelException: java.sql.SQLException: No suitable driver found for jdbc:TAOS-RS://localhost:6030stock?user=root&password=taosdata
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSinkWriter.<init>(TDengineSinkWriter.java:69)
at org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSink.createWriter(TDengineSink.java:52)
at org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSink.createWriter(TDengineSink.java:34)
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:272)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$15(SeaTunnelTask.java:420)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:417)
at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][tdengine] add cause while throwing TDengineConnectorException [seatunnel]
Posted by "alextinng (via GitHub)" <gi...@apache.org>.
alextinng closed issue #5998: [Improvement][tdengine] add cause while throwing TDengineConnectorException
URL: https://github.com/apache/seatunnel/issues/5998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [bug][tdengine] connector-tdengine didn't work cause no suitable driver found exception and NPE [seatunnel]
Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on issue #5998:
URL: https://github.com/apache/seatunnel/issues/5998#issuecomment-1868466240
@alextinng Look forward to your contributions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [bug][tdengine] connector-tdengine didn't work cause no suitable driver found exception and NPE [seatunnel]
Posted by "alextinng (via GitHub)" <gi...@apache.org>.
alextinng commented on issue #5998:
URL: https://github.com/apache/seatunnel/issues/5998#issuecomment-1914009052
> 这个问题解决了吗？ 我也遇到了类似的错误。
>
> ```
> 2024-01-28 16:35:02,200 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - report error from task
> org.apache.seatunnel.common.utils.SeaTunnelException: java.sql.SQLException: No suitable driver found for jdbc:TAOS-RS://localhost:6030stock?user=root&password=taosdata
> at java.sql.DriverManager.getConnection(DriverManager.java:689)
> at java.sql.DriverManager.getConnection(DriverManager.java:270)
> at org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSinkWriter.<init>(TDengineSinkWriter.java:69)
> at org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSink.createWriter(TDengineSink.java:52)
> at org.apache.seatunnel.connectors.seatunnel.tdengine.sink.TDengineSink.createWriter(TDengineSink.java:34)
> at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:272)
> at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$15(SeaTunnelTask.java:420)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:417)
> at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
> ```
see https://github.com/apache/seatunnel/pull/6088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org