You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2023/08/04 15:26:00 UTC
[jira] [Commented] (SPARK-44677) Drop legacy Hive-based ORC file format
[ https://issues.apache.org/jira/browse/SPARK-44677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751148#comment-17751148 ]
Dongjoon Hyun commented on SPARK-44677:
---------------------------------------
+1. Thank you, [~chengpan].
> Drop legacy Hive-based ORC file format
> --------------------------------------
>
> Key: SPARK-44677
> URL: https://issues.apache.org/jira/browse/SPARK-44677
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Cheng Pan
> Priority: Major
>
> Currently, Spark allows to use spark.sql.orc.impl=native/hive to switch the ORC FileFormat implementation.
> SPARK-23456(2.4) switched the default value of spark.sql.orc.impl from "hive" to "native". and prepared to drop the "hive" implementation in the future.
> > ... eventually, Apache Spark will drop old Hive-based ORC code.
> The native implementation works well during the whole Spark 3.x period, so it's a good time to consider dropping the "hive" one in Spark 4.0.
> Also, we should take care about the backward-compatibility during change.
> > BTW, IIRC, there was a different at Hive ORC CHAR implementation before. So, we couldn't remove it for backward-compatibility issues. Since Spark implements many CHAR features, we need to re-verify that {{native}} implementation has all legacy Hive-based ORC features
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org