[jira] [Resolved] (SPARK-5923) Very slow query when using Oracle hive metastore and table has lots of partitions

["Xiao Li (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Li resolved SPARK-5923.
----------------------------
    Resolution: Fixed

This should have been resolved. Please reopen it if this is still an issue. Thanks!

> Very slow query when using Oracle  hive metastore and table has lots of partitions
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-5923
>                 URL: https://issues.apache.org/jira/browse/SPARK-5923
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.2.0
>            Reporter: Matthew Taylor
>
> This has two aspects
> * The direct sql support for oracle is broken in hive 0.13.1. Fails when partitions get bigger than 1000 due oracle limitation on IN clause. This cause fall back to ORM which is very slow(20 minutes to even start the query)
> * Hive it self does not suffer this problem as it passes down to the metadata query, filter terms that restrict the partitions returned. SparkSQL is always asking for all partitions event if they are not all needed. Even when we patched hive it was still taking 2 minutes 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org