You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "NoPr (via GitHub)" <gi...@apache.org> on 2024/07/11 10:40:37 UTC
[I] [Bug] [Transformer] SQL -Not Supported CAST/CONVERT TIMESTAMP, DATE [seatunnel]
NoPr opened a new issue, #7172:
URL: https://github.com/apache/seatunnel/issues/7172
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
transform实现增量,因为transport不支持多源表,所以手动传入比较的时间值。
不匹配的左右值类型,
不能支持的CAST TYPE
### SeaTunnel Version
2.3.5
### SeaTunnel Config
```conf
"query " : "select col1 from fake_1810212286042378240_fa where cast(col1 as TIMESTAMP) > 1718208000"
"query " : "select col1 from fake_1810212286042378240_fa where col1 > 1718208000"
"query " : "select col1 from fake_1810212286042378240_fa where col1 > cast(1718208000 as timestamp)"
"query " : "select col1 from fake_1810212286042378240_fa where cast(col1 as Timestamp) > 1718208000"
col1 字段为mysql的date,尝试更改为timestamp还是同样的问题。
目标表为hive。增量值为手动获取
```
### Running Command
```shell
api提交
```
### Error Exception
```log
1.
{
"plugin_name": "Sql",
"result_table_name": "fake1_1810212286042378240_fa",
"source_table_name": "fake_1810212286042378240_fa",
"query " : "select col1 from fake_1810212286042378240_fa where col1 > cast(1718208000000 as DATE)"
}
此时报错:
org.apache.seatunnel.transform.exception.TransformException: ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - Unsupported CAST AS type: DATE
org.apache.seatunnel.transform.sql.zeta.functions.SystemFunction.castAs(SystemFunction.java:140)org.apache.seatunnel.transform.sql.zeta.ZetaSQLFunction.executeCastExpr(ZetaSQLFunction.java:532)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLFunction.computeForValue(ZetaSQLFunction.java:292)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLFilter.executeComparisonOperator(ZetaSQLFilter.java:258)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLFilter.executeFilter(ZetaSQLFilter.java:88)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLEngine.transformBySQL(ZetaSQLEngine.java:222)
org.apache.seatunnel.transform.sql.SQLTransform.transformRow(SQLTransform.java:115)
org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.map(AbstractSeaTunnelTransform.java:34)
org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.map(AbstractSeaTunnelTransform.java:25)
org.apache.seatunnel.engine.server.task.flow.TransformFlowLifeCycle.received(TransformFlowLifeCycle.java:94)
org.apache.seatunnel.engine.server.task.flow.TransformFlowLifeCycle.received(TransformFlowLifeCycle.java:37)
org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.sendRecordToNext(SeaTunnelSourceCollector.java:204)
org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:119)
org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceReader.pollNext(JdbcSourceReader.java:70)
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156)
org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116)
org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121)
org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703)
org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)"
假如传递参数为'2024-06-06 17:52:06'报相同的错误
2.
"query " : "select col1 from fake_1810212286042378240_fa where col1 > cast(1718208000 as DATE)"
此时报错:
java.time.DateTimeExceptin: Invalid value for MonthOfYear (valid values 1 - 12): 80
java.time.temporal.ValueRange.checkValidValue(ValueRange.java:311)
java.time.temporal.ChronoField.checkValidValue(ChrnoField.java:703)
java.time.LocalDate.of(LocalDate.java:267)
org.apache.seatunnel.transform.sql.zeta.functions.SystemFunction.castAs(SystemFunction.java:136)
org.apache.satunnel.transform.sql.zeta.ZetaSQLFunction.executeCastExpr(ZetaSQLFunction.java:532)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLFunction.computeForValue(ZetaSQLFunction.java:292)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLFilter.executeComparisonOperator(ZetaSQLFilter.java:258)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLFilter.executeFilter(ZetaQLFilter.java:88)
org.apache.seatunnel.transform.sql.zeta.ZetaSQLEngine.transformBySQL(ZetaSQLEngine.java:222)
org.apache.seatunnel.transform.sql.SQLTransform.transformRow(SQLTansform.java:115)
org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.map(AbstractSeaTunnelTransform.java:34)
org.apache.seatunnel.transform.common.AbstractSeaTunnlTransform.map(AbstractSeaTunnelTransform.java:25)
org.apache.seatunnel.engine.server.task.flow.TransformFlowLifeCycle.received(TransformFlowLifeCycle.java:94)
org.apache.seatunel.engine.server.task.flow.TransformFlowLifeCycle.received(TransformFlowLifeCycle.java:37)
org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.sendRecordToNext(SeaTunneSourceCollector.java:204)
org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:119)
org.apache.seatunnel.connectors.seatunnel.dbc.source.JdbcSourceReader.pollNext(JdbcSourceReader.java:70)
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156)
org.apacheseatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116)
org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121)
org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutinService.java:703)
org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)
java.util.concurrent.Executors$RunnableAdapter.cll(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.oncurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
3.
"query " : "select col1 from fake_1810212286042378240_fa where cast(col1 as TIMESTAMP) > 1718208000"
此时报错 :
org.apache.seatunnel.transform.exception.TransformException: ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - Filed types not matched, left is: LocalDateTime, right is: Integer
at org.apache.seatunnel.transform.sql.zeta.ZetaSQLFilter.greaterThanExpr(ZetaSQLFilter.java:309)
at org.apache.seatunnel.transform.sql.zeta.ZetaSQLFilter.executeFilter(ZetaSQLFilter.java:96)
at org.apache.seatunnel.transform.sql.zeta.ZetaSQLEngine.transformBySQL(ZetaSQLEngine.java:222)
at org.apache.seatunnel.transform.sql.SQLTransform.transformRow(SQLTransform.java:115)
at org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.map(AbstractSeaTunnelTransform.java:34)
at org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.map(AbstractSeaTunnelTransform.java:25)
at org.apache.seatunnel.engine.server.task.flow.TransformFlowLifeCycle.received(TransformFlowLifeCycle.java:94)
at org.apache.seatunnel.engine.server.task.flow.TransformFlowLifeCycle.received(TransformFlowLifeCycle.java:37)
at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.sendRecordToNext(SeaTunnelSourceCollector.java:204)
at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:119)
at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceReader.pollNext(JdbcSourceReader.java:70)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
不知道是我哪里出了问题
```
### Zeta or Flink or Spark Version
Zeta 2.3.5
### Java or Scala Version
jdk-17
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Transformer] SQL -Not Supported CAST/CONVERT TIMESTAMP, DATE [seatunnel]
Posted by "NoPr (via GitHub)" <gi...@apache.org>.
NoPr closed issue #7172: [Bug] [Transformer] SQL -Not Supported CAST/CONVERT TIMESTAMP, DATE
URL: https://github.com/apache/seatunnel/issues/7172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Transformer] SQL -Not Supported CAST/CONVERT TIMESTAMP, DATE [seatunnel]
Posted by "NoPr (via GitHub)" <gi...@apache.org>.
NoPr commented on issue #7172:
URL: https://github.com/apache/seatunnel/issues/7172#issuecomment-2224302583
"transform": [
{
"plugin_name": "Sql",
"result_table_name": "fake1_1810212286042378241_fa",
"source_table_name": "fake_1810212286042378240_fa",
"query " : "select cast(col1 as TimeStamp)as col1 from fake_1810212286042378240_fa "
},
{
"plugin_name": "Sql",
"result_table_name": "fake1_1810212286042378242_fa",
"source_table_name": "fake1_1810212286042378241_fa",
"query " : "select col1 from fake1_1810212286042378241_fa where col1 > 1058233939"
},
多段transform依然报错左右字段类型不匹配
{
"plugin_name": "Sql",
"result_table_name": "fake1_1810212286042378241_fa",
"source_table_name": "fake_1810212286042378240_fa",
"query " : "select cast(col1 as Integer)as col1 from fake_1810212286042378240_fa "
},
{
"plugin_name": "Sql",
"result_table_name": "fake1_1810212286042378242_fa",
"source_table_name": "fake1_1810212286042378241_fa",
"query " : "select col1 from fake1_1810212286042378241_fa where col1 > 1058233939"
},这个报错"java.lang.NumberFormatException: For input string: \"2024-06-13T00:00
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Transformer] SQL -Not Supported CAST/CONVERT TIMESTAMP, DATE [seatunnel]
Posted by "NoPr (via GitHub)" <gi...@apache.org>.
NoPr commented on issue #7172:
URL: https://github.com/apache/seatunnel/issues/7172#issuecomment-2224902010
@Hisoka-X 你好,这和问题可以麻烦关闭一下吗?
"query " : "select col1 from fake_1810212286042378240_fa where col1 > TO_DATE('2024-06-13 02:22:13','yyyy-MM-dd HH:mm:ss')"
我使用来了新的函数来处理时间戳,已经解决问题了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org