You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2019/09/05 07:47:00 UTC
[jira] [Assigned] (SPARK-28878) DataSourceV2 should not insert
extra projection for columnar batches
[ https://issues.apache.org/jira/browse/SPARK-28878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-28878:
-----------------------------------
Assignee: Ryan Blue
> DataSourceV2 should not insert extra projection for columnar batches
> --------------------------------------------------------------------
>
> Key: SPARK-28878
> URL: https://issues.apache.org/jira/browse/SPARK-28878
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Ryan Blue
> Assignee: Ryan Blue
> Priority: Major
>
> SPARK-23325 added an extra physical projection when reading from a DSv2 source because some Spark operators assume that InternalRow instances are actually UnsafeRow. The projection ensures that InternalRow is converted to UnsafeRow. This isn't needed for the columnar batch read path because this is already done when converting from columnar operators to row-based operators in InputRDDCodegen.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org