You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/09/26 03:47:00 UTC
[jira] [Resolved] (SPARK-36854) Parquet reader fails on load of
ANSI interval when off-heap is enabled
[ https://issues.apache.org/jira/browse/SPARK-36854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-36854.
----------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 34106
[https://github.com/apache/spark/pull/34106]
> Parquet reader fails on load of ANSI interval when off-heap is enabled
> ----------------------------------------------------------------------
>
> Key: SPARK-36854
> URL: https://issues.apache.org/jira/browse/SPARK-36854
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Fix For: 3.3.0
>
>
> When off-heap column vector is enabled, parquet reader fails. The example below portraits the issue:
> {code:scala}
> scala> Seq(java.time.Period.of)
> scala> val df = Seq(java.time.Period.ofMonths(1)).toDF
> df: org.apache.spark.sql.DataFrame = [value: interval year to month]
> scala> df.write.parquet("/Users/maximgekk/tmp/parquet_offheap")
> scala> spark.conf.set("spark.sql.columnVector.offheap.enabled", true)
> scala> spark.read.parquet("/Users/maximgekk/tmp/parquet_offheap")
> res2: org.apache.spark.sql.DataFrame = [value: interval year to month]
> scala> spark.read.parquet("/Users/maximgekk/tmp/parquet_offheap").show()
> 21/09/25 22:09:03 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
> java.lang.RuntimeException: Unhandled YearMonthIntervalType(0,1)
> at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.reserveInternal(OffHeapColumnVector.java:562)
> at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.<init>(OffHeapColumnVector.java:75)
> at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.allocateColumns(OffHeapColumnVector.java:53)
> at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.allocateColumns(OffHeapColumnVector.java:42)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org