You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by "ying-hua (via GitHub)" <gi...@apache.org> on 2024/09/28 07:36:08 UTC
[I] [Bug][SDK] Limitation of Transform Sql Statement Parser [inlong]
ying-hua opened a new issue, #11228:
URL: https://github.com/apache/inlong/issues/11228
### What happened
Transform sql cannot support parse long attribute chains after accessing array, such as "$child(0).msg.msgId"
### Error Log
```
net.sf.jsqlparser.JSQLParserException: Encountered unexpected token: "," ","
at line 1, column 37.
Was expecting one of:
"("
at net.sf.jsqlparser.parser.CCJSqlParserManager.parse(CCJSqlParserManager.java:25)
at org.apache.inlong.sdk.transform.process.TransformProcessor.initTransformSql(TransformProcessor.java:102)
at org.apache.inlong.sdk.transform.process.TransformProcessor.init(TransformProcessor.java:94)
at org.apache.inlong.sdk.transform.process.TransformProcessor.<init>(TransformProcessor.java:83)
at org.apache.inlong.sdk.transform.process.TransformProcessor.create(TransformProcessor.java:72)
at org.apache.inlong.sdk.transform.process.processor.TestJson2CsvProcessor.testJson2CsvForBug(TestJson2CsvProcessor.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: net.sf.jsqlparser.parser.ParseException: Encountered unexpected token: "," ","
at line 1, column 37.
Was expecting one of:
"("
at net.sf.jsqlparser.parser.CCJSqlParser.generateParseException(CCJSqlParser.java:38866)
at net.sf.jsqlparser.parser.CCJSqlParser.jj_consume_token(CCJSqlParser.java:38696)
at net.sf.jsqlparser.parser.CCJSqlParser.InternalFunction(CCJSqlParser.java:17060)
at net.sf.jsqlparser.parser.CCJSqlParser.Function(CCJSqlParser.java:16986)
at net.sf.jsqlparser.parser.CCJSqlParser.InternalFunction(CCJSqlParser.java:17154)
at net.sf.jsqlparser.parser.CCJSqlParser.Function(CCJSqlParser.java:16986)
at net.sf.jsqlparser.parser.CCJSqlParser.PrimaryExpression(CCJSqlParser.java:13961)
at net.sf.jsqlparser.parser.CCJSqlParser.BitwiseXor(CCJSqlParser.java:13763)
at net.sf.jsqlparser.parser.CCJSqlParser.MultiplicativeExpression(CCJSqlParser.java:13716)
at net.sf.jsqlparser.parser.CCJSqlParser.AdditiveExpression(CCJSqlParser.java:13679)
at net.sf.jsqlparser.parser.CCJSqlParser.BitwiseAndOr(CCJSqlParser.java:13625)
at net.sf.jsqlparser.parser.CCJSqlParser.ConcatExpression(CCJSqlParser.java:13601)
at net.sf.jsqlparser.parser.CCJSqlParser.SimpleExpression(CCJSqlParser.java:13587)
at net.sf.jsqlparser.parser.CCJSqlParser.SQLCondition(CCJSqlParser.java:12761)
at net.sf.jsqlparser.parser.CCJSqlParser.Condition(CCJSqlParser.java:12491)
at net.sf.jsqlparser.parser.CCJSqlParser.SelectExpressionItem(CCJSqlParser.java:8561)
at net.sf.jsqlparser.parser.CCJSqlParser.SelectItem(CCJSqlParser.java:8597)
at net.sf.jsqlparser.parser.CCJSqlParser.SelectItemsList(CCJSqlParser.java:8551)
at net.sf.jsqlparser.parser.CCJSqlParser.PlainSelect(CCJSqlParser.java:7838)
at net.sf.jsqlparser.parser.CCJSqlParser.SetOperationList(CCJSqlParser.java:8107)
at net.sf.jsqlparser.parser.CCJSqlParser.SelectBody(CCJSqlParser.java:7718)
at net.sf.jsqlparser.parser.CCJSqlParser.Select(CCJSqlParser.java:7713)
at net.sf.jsqlparser.parser.CCJSqlParser.SingleStatement(CCJSqlParser.java:377)
at net.sf.jsqlparser.parser.CCJSqlParser.Statement(CCJSqlParser.java:285)
at net.sf.jsqlparser.parser.CCJSqlParserManager.parse(CCJSqlParserManager.java:23)
... 32 more
Process finished with exit code 255
```
### What you expected to happen
This sql statement should be parsed normally.
### How to reproduce
Add this test function to TestJson2CsvProcessor.java and run.
```java
@Test
public void testJson2CsvForBug() throws Exception {
List<FieldInfo> fields = this.getTestFieldList("sid", "msgId", "msgBody");
JsonSourceInfo jsonSource = new JsonSourceInfo("UTF-8", "msgs");
CsvSinkInfo csvSink = new CsvSinkInfo("UTF-8", '|', '\\', fields);
String transformSql = "select $root.sid,$child.msg.msgId,$child.msg.msgBody from source";
TransformConfig config = new TransformConfig(transformSql);
// case1 with no bug
TransformProcessor<String, String> processor = TransformProcessor
.create(config, SourceDecoderFactory.createJsonDecoder(jsonSource),
SinkEncoderFactory.createCsvEncoder(csvSink));
String srcString = "{\n" +
" \"sid\": \"value1\",\n" +
" \"msgs\": [\n" +
" {\"msg\":{ \"msgId\": \"111\", \"msgBody\": \"msg1_body\"}},\n" +
" {\"msg\":{ \"msgId\": \"222\", \"msgBody\": \"msg2_body\"}}\n" +
" ]\n" +
"}";
List<String> output = processor.transform(srcString, new HashMap<>());
Assert.assertEquals(2, output.size());
Assert.assertEquals(output.get(0), "value1|111|msg1_body");
Assert.assertEquals(output.get(1), "value1|222|msg2_body");
// case2 with bug
transformSql = "select $root.sid,$child(0).msg.msgId,$child(1).msg.msgBody from source";
config = new TransformConfig(transformSql);
processor = TransformProcessor
.create(config, SourceDecoderFactory.createJsonDecoder(jsonSource),
SinkEncoderFactory.createCsvEncoder(csvSink));
srcString = "{\n" +
" \"sid\": \"value1\",\n" +
" \"msgs\": [\n" +
" {\"msg\":{ \"msgId\": \"111\", \"msgBody\": \"msg1_body\"}},\n" +
" {\"msg\":{ \"msgId\": \"222\", \"msgBody\": \"msg2_body\"}}\n" +
" ]\n" +
"}";
output = processor.transform(srcString, new HashMap<>());
Assert.assertEquals(1, output.size());
Assert.assertEquals(output.get(0), "value1|111|msg2_body");
}
```
### Environment
_No response_
### InLong version
master
### InLong Component
InLong SDK
### Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][SDK] Limitation of Transform Sql Statement Parser [inlong]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11228:
URL: https://github.com/apache/inlong/issues/11228#issuecomment-2537594181
This issue is stale because it has been open for 60 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][SDK] Limitation of Transform Sql Statement Parser [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on issue #11228:
URL: https://github.com/apache/inlong/issues/11228#issuecomment-2606492397
The Tencent Rhino-bird 2024 is finished, and this issue will be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][SDK] Limitation of Transform Sql Statement Parser [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng closed issue #11228: [Bug][SDK] Limitation of Transform Sql Statement Parser
URL: https://github.com/apache/inlong/issues/11228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org