[jira] [Commented] (CALCITE-1894) CsvTest.testCsvStream failing often

["Vladimir Sitnikov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616688#comment-16616688 ] 

Vladimir Sitnikov commented on CALCITE-1894:
--------------------------------------------

The issue is not a timeout.
The issue is CsvTest.testCsvStream does not perform what it is intended to do.

1) CsvStreamReader constructor has Thread.sleep. I don't think it is acceptable: https://github.com/apache/calcite/blob/c12e37d41a9d81b1303a345bad5067e1518d6c7d/example/csv/src/main/java/org/apache/calcite/adapter/csv/CsvStreamReader.java#L88

2) Linq4j.EnumeratorIterator calls {{moveNext}} in the constructor, which effectively fetches the first element.
It causes early fetch of the results.

https://github.com/apache/calcite/blob/d59b639d27da704f00eff616324a2c04aa06f84c/linq4j/src/main/java/org/apache/calcite/linq4j/Linq4j.java#L669-L676

I suggest to use something like
{code:java}    Boolean hasNext;

    EnumeratorIterator(Enumerator<T> enumerator) {
      this.enumerator = enumerator;
    }

    public boolean hasNext() {
      if (hasNext == null) {
        hasNext = enumerator.moveNext();
      }
      return hasNext;
    }

    public T next() {
      if (hasNext == null) {
        enumerator.moveNext();
      }
      T t = enumerator.current();
      hasNext = enumerator.moveNext();
      return t;
    }
{code}

If I correct the above issues, the following exception appears:
{noformat}java.lang.NumberFormatException: For input string: "DEPTNO:int"

	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.base/java.lang.Integer.parseInt(Integer.java:652)
	at java.base/java.lang.Integer.parseInt(Integer.java:770)
	at org.apache.calcite.adapter.csv.CsvEnumerator$RowConverter.convert(CsvEnumerator.java:273)
	at org.apache.calcite.adapter.csv.CsvEnumerator$ArrayRowConverter.convertStreamRow(CsvEnumerator.java:367)
	at org.apache.calcite.adapter.csv.CsvEnumerator$ArrayRowConverter.convertRow(CsvEnumerator.java:347)
	at org.apache.calcite.adapter.csv.CsvEnumerator$ArrayRowConverter.convertRow(CsvEnumerator.java:327)
	at org.apache.calcite.adapter.csv.CsvEnumerator.moveNext(CsvEnumerator.java:214)
	at org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1875)
	at org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1875)
	at org.apache.calcite.linq4j.TransformedEnumerator.moveNext(TransformedEnumerator.java:35)
	at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.hasNext(Linq4j.java:680)
	at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:45)
	at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
	at org.apache.calcite.test.CsvTest.testCsvStream(CsvTest.java:903)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:844)
{noformat}

It looks like it does not detect CSV header, and it tries to parse the first row as data.

I've no idea how CSV identifies if it should throw-away the first line or not, however the above exception does look like a true issue.

> CsvTest.testCsvStream failing often
> -----------------------------------
>
>                 Key: CALCITE-1894
>                 URL: https://issues.apache.org/jira/browse/CALCITE-1894
>             Project: Calcite
>          Issue Type: Bug
>            Reporter: Rajeshbabu Chintaguntla
>            Assignee: Julian Hyde
>            Priority: Major
>
> CsvTest.testCsvStream is timing out and failing often.
> This is the build failed because of this test.
> https://travis-ci.org/apache/calcite/jobs/253794426
> {noformat}
> <<< FAILURE! - in org.apache.calcite.test.CsvTest
> testCsvStream(org.apache.calcite.test.CsvTest)  Time elapsed: 10.044 sec  <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 10000 milliseconds
> 	at org.apache.calcite.test.CsvTest.testCsvStream(CsvTest.java:729)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)