You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2017/05/09 19:11:04 UTC
[jira] [Comment Edited] (SPARK-20682) Support a new faster ORC data
source based on Apache ORC
[ https://issues.apache.org/jira/browse/SPARK-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003326#comment-16003326 ]
Dongjoon Hyun edited comment on SPARK-20682 at 5/9/17 7:10 PM:
---------------------------------------------------------------
I fixed the description. Yep. `hive-storage-api` was a old huddle before 1.4.0. 1.4.0 provides `nohive` artifacts, too.
https://repo.maven.apache.org/maven2/org/apache/orc/orc-core/1.4.0/
I'll make a PR soon as a POC.
was (Author: dongjoon):
Yep. That was a old huddle before 1.4.0. 1.4.0 provides `nohive` artifacts, too.
https://repo.maven.apache.org/maven2/org/apache/orc/orc-core/1.4.0/
I'll make a PR soon as a POC.
> Support a new faster ORC data source based on Apache ORC
> --------------------------------------------------------
>
> Key: SPARK-20682
> URL: https://issues.apache.org/jira/browse/SPARK-20682
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1, 1.5.2, 1.6.3, 2.1.1
> Reporter: Dongjoon Hyun
>
> Since SPARK-2883, Apache Spark supports Apache ORC inside `sql/hive` module with Hive dependency. This issue aims to add a new and faster ORC data source inside `sql/core` and to replace the old ORC data source eventually. In this issue, the latest Apache ORC 1.4.0 (released yesterday) is used.
> There are four key benefits.
> - Speed: Use both Spark `ColumnarBatch` and ORC `RowBatch` together. This is faster than the current implementation in Spark.
> - Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more.
> - Usability: User can use `ORC` data sources without hive module, i.e, `-Phive`.
> - Maintainability: Reduce the Hive dependency and can remove old legacy code later.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org