You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Chiradip Mandal (JIRA)" <ji...@apache.org> on 2019/02/06 02:52:00 UTC
[jira] [Comment Edited] (CALCITE-2769) CSV Adapter does not handle
- Empty and malformed csv lines, space before or after comma
[ https://issues.apache.org/jira/browse/CALCITE-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761387#comment-16761387 ]
Chiradip Mandal edited comment on CALCITE-2769 at 2/6/19 2:51 AM:
------------------------------------------------------------------
How to reproduce:
Use this as DEPTS.csv content
DEPTNO:int,NAME:string
DEPTNO:int,NAME:string
10,"Sales"
20,"Marketing"
30,"Accounts"
31,"My Dept"
32, "My Another Depts"
Run "mvn clean test -DskipGenerate -pl example/csv"
Test fails with this message:
[*ERROR*] *Tests* *run: 39*, Failures: 0, *Errors: 5*, *Skipped: 3*, Time elapsed: 2.337 s *<<< FAILURE!* - in org.apache.calcite.test.*CsvTest*
[*ERROR*] testPrepared(org.apache.calcite.test.CsvTest) Time elapsed: 0.152 s <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.collect(CsvTest.java:374)
at org.apache.calcite.test.CsvTest.lambda$expect$1(CsvTest.java:320)
at org.apache.calcite.test.CsvTest.testPrepared(CsvTest.java:691)
[*ERROR*] testJoinOnString(org.apache.calcite.test.CsvTest) Time elapsed: 0.055 s <<< ERROR!
java.lang.RuntimeException:
java.sql.SQLException: Error while executing SQL "select * from emps
join depts on emps.name = depts.name": 1
at org.apache.calcite.test.CsvTest.testJoinOnString(CsvTest.java:409)
Caused by: java.sql.SQLException:
Error while executing SQL "select * from emps
join depts on emps.name = depts.name": 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testJoinOnString(CsvTest.java:409)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testJoinOnString(CsvTest.java:409)
[*ERROR*] testSelectSingleProject(org.apache.calcite.test.CsvTest) Time elapsed: 0.017 s <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.output(CsvTest.java:393)
at org.apache.calcite.test.CsvTest.output(CsvTest.java:961)
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:357)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testSelectSingleProject(CsvTest.java:185)
[*ERROR*] testReadme(org.apache.calcite.test.CsvTest) Time elapsed: 0.1 s <<< ERROR!
java.lang.RuntimeException: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.testReadme(CsvTest.java:471)
Caused by: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testReadme(CsvTest.java:471)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testReadme(CsvTest.java:471)
[*ERROR*] testBlankLinesAndSpaces(org.apache.calcite.test.CsvTest) Time elapsed: 0.067 s <<< ERROR!
java.lang.RuntimeException: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.testBlankLinesAndSpaces(CsvTest.java:480)
Caused by: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testBlankLinesAndSpaces(CsvTest.java:480)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testBlankLinesAndSpaces(CsvTest.java:480)
[*INFO*]
[*INFO*] Results:
[*INFO*]
[*ERROR*] *Errors:*
[*ERROR*] *CsvTest.testBlankLinesAndSpaces:480 » Runtime java.sql.SQLException: Error whi...*
[*ERROR*] *CsvTest.testJoinOnString:409 » Runtime java.sql.SQLException: Error while exec...*
[*ERROR*] *CsvTest.testPrepared:691->lambda$expect$1:320->collect:374 » ArrayIndexOutOfBounds*
[*ERROR*] *CsvTest.testReadme:471 » Runtime java.sql.SQLException: Error while executing ...*
[*ERROR*] *CsvTest.testSelectSingleProject:185->access$300:67->checkSql:357->output:961->output:393 » ArrayIndexOutOfBounds*
[*INFO*]
[*ERROR*] *Tests run: 39, Failures: 0, Errors: 5, Skipped: 3*
[*INFO*]
[*INFO*] *------------------------------------------------------------------------*
[*INFO*] *BUILD FAILURE*
was (Author: chiradip):
How to reproduce:
Use this as DEPTS.csv content
DEPTNO:int,NAME:string
DEPTNO:int,NAME:string
10,"Sales"
20,"Marketing"
30,"Accounts"
31,"My Dept"
32, "My Another Depts"
Run "mvn clean test -DskipGenerate -pl example/csv"
Test fails with this message:
[*ERROR*] *Tests* *run: 39*, Failures: 0, *Errors: 5*, *Skipped: 3*, Time elapsed: 2.337 s *<<< FAILURE!* - in org.apache.calcite.test.*CsvTest*
[*ERROR*] testPrepared(org.apache.calcite.test.CsvTest) Time elapsed: 0.152 s <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.collect(CsvTest.java:374)
at org.apache.calcite.test.CsvTest.lambda$expect$1(CsvTest.java:320)
at org.apache.calcite.test.CsvTest.testPrepared(CsvTest.java:691)
[*ERROR*] testJoinOnString(org.apache.calcite.test.CsvTest) Time elapsed: 0.055 s <<< ERROR!
java.lang.RuntimeException:
java.sql.SQLException: Error while executing SQL "select * from emps
join depts on emps.name = depts.name": 1
at org.apache.calcite.test.CsvTest.testJoinOnString(CsvTest.java:409)
Caused by: java.sql.SQLException:
Error while executing SQL "select * from emps
join depts on emps.name = depts.name": 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testJoinOnString(CsvTest.java:409)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testJoinOnString(CsvTest.java:409)
[*ERROR*] testSelectSingleProject(org.apache.calcite.test.CsvTest) Time elapsed: 0.017 s <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.output(CsvTest.java:393)
at org.apache.calcite.test.CsvTest.output(CsvTest.java:961)
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:357)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testSelectSingleProject(CsvTest.java:185)
[*ERROR*] testReadme(org.apache.calcite.test.CsvTest) Time elapsed: 0.1 s <<< ERROR!
java.lang.RuntimeException: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.testReadme(CsvTest.java:471)
Caused by: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testReadme(CsvTest.java:471)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testReadme(CsvTest.java:471)
[*ERROR*] testBlankLinesAndSpaces(org.apache.calcite.test.CsvTest) Time elapsed: 0.067 s <<< ERROR!
java.lang.RuntimeException: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.testBlankLinesAndSpaces(CsvTest.java:480)
Caused by: java.sql.SQLException: Error while executing SQL "SELECT d.name, COUNT(*) cnt FROM emps AS e JOIN depts AS d ON e.deptno = d.deptno GROUP BY d.name": 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testBlankLinesAndSpaces(CsvTest.java:480)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.calcite.test.CsvTest.checkSql(CsvTest.java:355)
at org.apache.calcite.test.CsvTest.access$300(CsvTest.java:67)
at org.apache.calcite.test.CsvTest.testBlankLinesAndSpaces(CsvTest.java:480)
[*INFO*]
[*INFO*] Results:
[*INFO*]
[*ERROR*] *Errors:*
[*ERROR*] *CsvTest.testBlankLinesAndSpaces:480 » Runtime java.sql.SQLException: Error whi...*
[*ERROR*] *CsvTest.testJoinOnString:409 » Runtime java.sql.SQLException: Error while exec...*
[*ERROR*] *CsvTest.testPrepared:691->lambda$expect$1:320->collect:374 » ArrayIndexOutOfBounds*
[*ERROR*] *CsvTest.testReadme:471 » Runtime java.sql.SQLException: Error while executing ...*
[*ERROR*] *CsvTest.testSelectSingleProject:185->access$300:67->checkSql:357->output:961->output:393 » ArrayIndexOutOfBounds*
[*INFO*]
[*ERROR*] *Tests run: 39, Failures: 0, Errors: 5, Skipped: 3*
[*INFO*]
[*INFO*] *------------------------------------------------------------------------*
[*INFO*] *BUILD FAILURE*
> CSV Adapter does not handle - Empty and malformed csv lines, space before or after comma
> ----------------------------------------------------------------------------------------
>
> Key: CALCITE-2769
> URL: https://issues.apache.org/jira/browse/CALCITE-2769
> Project: Calcite
> Issue Type: Bug
> Components: csv-adapter
> Affects Versions: 1.18.0
> Reporter: Chiradip Mandal
> Assignee: Julian Hyde
> Priority: Major
> Fix For: next
>
>
> File: org.apache.calcite.adapter.csv.CsvEnumerator.java in example folder did not handle CSV with space between delimiter i.e."," and also did not handle empty lines or malformed lines in the middle of the file or to the end. The sqlline crashes is there is an empty line with java.lang.ArrayIndexOutOfBoundsException.
>
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.calcite.adapter.csv.CsvEnumerator$ArrayRowConverter.convertNormalRow(CsvEnumerator.java:347)
> at org.apache.calcite.adapter.csv.CsvEnumerator$ArrayRowConverter.convertRow(CsvEnumerator.java:339)
> at org.apache.calcite.adapter.csv.CsvEnumerator$ArrayRowConverter.convertRow(CsvEnumerator.java:317)
> at org.apache.calcite.adapter.csv.CsvEnumerator.moveNext(CsvEnumerator.java:204)
> at org.apache.calcite.linq4j.EnumerableDefaults$10$1.moveNext(EnumerableDefaults.java:1878)
> at org.apache.calcite.linq4j.EnumerableDefaults$10$1.moveNext(EnumerableDefaults.java:1878)
> at org.apache.calcite.linq4j.TransformedEnumerator.moveNext(TransformedEnumerator.java:35)
> at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.next(Linq4j.java:684)
> at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:38)
> at sqlline.SqlLine.print(SqlLine.java:1719)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.begin(SqlLine.java:540)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)