You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/07/26 23:39:04 UTC
[jira] [Commented] (SPARK-9332) CatalystTypeConverters.toScala does
not work on UnsafeRows
[ https://issues.apache.org/jira/browse/SPARK-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642102#comment-14642102 ]
Reynold Xin commented on SPARK-9332:
------------------------------------
Ah yes - need to add the if branches. Should be trivial to fix those now though.
> CatalystTypeConverters.toScala does not work on UnsafeRows
> ----------------------------------------------------------
>
> Key: SPARK-9332
> URL: https://issues.apache.org/jira/browse/SPARK-9332
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Josh Rosen
> Assignee: Reynold Xin
> Priority: Critical
> Fix For: 1.5.0
>
>
> When CatalystTypeConverters.toScala is applied to an UnsafeRow it may lead to an UnsupportedOperationException when it tries to use a generic getter to retrieve primitive columns.
> For example:
> {code}
> == Parsed Logical Plan ==
> Sort [c183#208 ASC,c183#208 ASC], true
> Project [c183#208,c184#209]
> LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> == Analyzed Logical Plan ==
> c183: binary, c184: int
> Sort [c183#208 ASC,c183#208 ASC], true
> Project [c183#208,c184#209]
> LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> == Optimized Logical Plan ==
> Sort [c183#208 ASC,c183#208 ASC], true
> LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> == Physical Plan ==
> UnsafeExternalSort [c183#208 ASC,c183#208 ASC], true, 0
> Exchange rangepartitioning(c183#208 ASC,c183#208 ASC)
> PhysicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> {code}
> {code}
> java.lang.UnsupportedOperationException
> at org.apache.spark.sql.catalyst.expressions.UnsafeRow.get(UnsafeRow.java:223)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$IdentityConverter$.toScalaImpl(CatalystTypeConverters.scala:143)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:118)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:266)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:233)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:386)
> at org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
> at org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org