[jira] [Commented] (TOREE-531) notebook failures does not print whole stack trace for spark session DF

["SHOBHIT SHUKLA (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17511916#comment-17511916 ] 

SHOBHIT SHUKLA commented on TOREE-531:
--------------------------------------

Refer below example to reproduce :
The simple example below shows that in case of any syntax error, the Scala 2.12 with Spark Notebook doesn’t throw error the way it is thrown in the Python 3.7 Notebook. Removed a double quote (“) in the end during the assignment to the variable test2 to showcase this issue.
 !f5b22480-8a67-11ec-88d3-93fd083c850b.png! 

> notebook failures does not print whole stack trace for spark session DF
> -----------------------------------------------------------------------
>
>                 Key: TOREE-531
>                 URL: https://issues.apache.org/jira/browse/TOREE-531
>             Project: TOREE
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 0.4.0, 0.5.0
>            Reporter: SHOBHIT SHUKLA
>            Priority: Major
>         Attachments: f5b22480-8a67-11ec-88d3-93fd083c850b.png
>
>
> The following stack trace is printed which is not whole stack trace, it is missing the lines in the beginning
> at wdpshadow.org.apache.spark.sql.util.ArrowUtils$.fromArrowType(ArrowUtils.scala:75)
> at wdpshadow.org.apache.spark.sql.util.ArrowUtils$.fromArrowField(ArrowUtils.scala:125)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$.createFieldWriter(ArrowWriter.scala:48)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:41)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$$$Lambda$2701/0x0000000075bfb4a0.apply(Unknown Source)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.TraversableLike$$Lambda$34/0x00000000f27b9810.apply(Unknown Source)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$.create(ArrowWriter.scala:39)
> at com.ibm.connect.spark.arrow.AbstractArrowDataSource.inferSchema(AbstractArrowDataSource.scala:65)
> at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:81)
> at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:274)
> at org.apache.spark.sql.DataFrameReader$$Lambda$2635/0x00000000756d4b20.apply(Unknown Source)
> at scala.Option.map(Option.scala:230)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:248)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:221)
> *Expected:*
> java.lang.UnsupportedOperationException: Unsupported data type: Timestamp(MILLISECOND, null)
> at wdpshadow.org.apache.spark.sql.util.ArrowUtils$.fromArrowType(ArrowUtils.scala:75)
> at wdpshadow.org.apache.spark.sql.util.ArrowUtils$.fromArrowField(ArrowUtils.scala:125)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$.createFieldWriter(ArrowWriter.scala:48)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:41)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$$$Lambda$2822/0x000000005d1c9070.apply(Unknown Source)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.TraversableLike$$Lambda$34/0x00000000de7979c0.apply(Unknown Source)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at wdpshadow.org.apache.spark.sql.execution.arrow.ArrowWriter$.create(ArrowWriter.scala:39)
> at com.ibm.connect.spark.arrow.AbstractArrowDataSource.inferSchema(AbstractArrowDataSource.scala:65)
> at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:81)
> at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:274)
> at org.apache.spark.sql.DataFrameReader$$Lambda$2787/0x000000005d86a2f0.apply(Unknown Source)
> at scala.Option.map(Option.scala:230)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:248)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:221)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)