You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sunil Srivatsa (JIRA)" <ji...@apache.org> on 2016/10/21 22:52:58 UTC
[jira] [Created] (SPARK-18059) Spark not respecting partitions in
partitioned Hive views
Sunil Srivatsa created SPARK-18059:
--------------------------------------
Summary: Spark not respecting partitions in partitioned Hive views
Key: SPARK-18059
URL: https://issues.apache.org/jira/browse/SPARK-18059
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 1.5.1
Environment: ResourceManager version: 2.6.0-cdh5.4.2 from 15b703c8725733b7b2813d2325659eb7d57e7a3f by jenkins source checksum e7a085479aa1989b5cecfabea403549 on 2015-05-20T00:05Z
Hadoop version: 2.6.0-cdh5.4.2 from 15b703c8725733b7b2813d2325659eb7d57e7a3f by jenkins source checksum de74f1adb3744f8ee85d9a5b98f90d on 2015-05-19T23:58Z
Reporter: Sunil Srivatsa
For Hive partitioned views (https://cwiki.apache.org/confluence/display/Hive/PartitionedViews), when you specify a partition Spark reads from all partitions rather than just the specified ones. For example:
create table srcpart_1 (key int, value string) partitioned by (ds string);
create table srcpart_2 (key int, value string) partitioned by (ds string);
CREATE VIEW vp2
PARTITIONED ON (ds)
AS
SELECT srcpart_1.key, srcpart_2.value, srcpart_1.ds
FROM srcpart_1
join srcpart_2
on srcpart_1.key = srcpart_2.key
and srcpart_1.ds=srcpart_2.ds;
ALTER VIEW vp2 ADD PARTITION (ds='2016-10-01')
When a query executes, e.g.:
SELECT key, value FROM vp2 WHERE ds = '2016-01-01'
Spark reads Parquet files for all partitions. This is easy to observe in a job by printing out df.inputFiles. The Hive explains for executing that query against the view are the same as a normal join, which suggests Hive is behaving correctly:
SELECT srcpart_1.key, srcpart_2.value, srcpart_1.ds FROM srcpart_1
JOIN srcpart_2 ON srcpart_1.key = srcpart_2.key WHERE srcpart_1.ds = '2016-01-01' and srcpart_2.ds = '2016-01-01'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org