You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "baishuo (JIRA)" <ji...@apache.org> on 2014/08/13 11:09:12 UTC
[jira] [Comment Edited] (SPARK-3007) Add "Dynamic Partition"
support to Spark Sql hive
[ https://issues.apache.org/jira/browse/SPARK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095300#comment-14095300 ]
baishuo edited comment on SPARK-3007 at 8/13/14 9:08 AM:
---------------------------------------------------------
after modify the code, I can run the hiveql with dynamic partition by SparkSqlCLIDriver:
spark-sql> insert overwrite table partition_test_spark partition(stat_date,province)
select member_id2,name2,stat_date2,province2
from partition_test_input_spark2;
spark-sql>Time taken: 10.351 seconds
spark-sql>select * from partition_test_spark;
1 11 date1 pr1
2 22 date1 pr1
3 33 date1 pr2
4 44 date1 pr2
5 55 date2 pr1
6 66 date2 pr1
7 77 date2 pr2
8 88 date2 pr2
spark-sql> Time taken: 0.287 seconds
spark-sql>insert overwrite table partition_test_spark partition(stat_date='date1',province)
select member_id2,name2,province2
from partition_test_input_spark2
where stat_date2='date2';
spark-sql>select * from partition_test_spark;
5 55 date1 pr1
6 66 date1 pr1
7 77 date1 pr2
8 88 date1 pr2
5 55 date2 pr1
6 66 date2 pr1
7 77 date2 pr2
8 88 date2 pr2
and we can also check that data all located in exceped directionary
----------------------------------------------------------------------
the script to create partition_test_input_spark2 and
create table partition_test_input_spark2
(member_id2 string,
name2 string,
stat_date2 string,
province2 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
LOAD DATA LOCAL INPATH '/root/Desktop/testpartition.txt' OVERWRITE INTO TABLE partition_test_input_spark2;
(the content of testpartition.txt
is:
1,11,date1,pr1
2,22,date1,pr1
3,33,date1,pr2
4,44,date1,pr2
5,55,date2,pr1
6,66,date2,pr1
7,77,date2,pr2
8,88,date2,pr2)
create table partition_test_spark
(member_id string,
name string
)
partitioned by (
stat_date string,
province string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
was (Author: baishuo):
after modify the code, I can run the hiveql with dynamic partition by SparkSqlCLIDriver:
spark-sql> insert overwrite table partition_test_spark partition(stat_date,province)
select member_id2,name2,stat_date2,province2
from partition_test_input_spark2;
spark-sql>Time taken: 10.351 seconds
spark-sql>select * from partition_test_spark;
1 11 date1 pr1
2 22 date1 pr1
3 33 date1 pr2
4 44 date1 pr2
5 55 date2 pr1
6 66 date2 pr1
7 77 date2 pr2
8 88 date2 pr2
spark-sql> Time taken: 0.287 seconds
spark-sql>insert overwrite table partition_test_spark partition(stat_date='date1',province)
select member_id2,name2,province2
from partition_test_input_spark2
where stat_date2='date2';
spark-sql>select * from partition_test_spark;
5 55 date1 pr1
6 66 date1 pr1
7 77 date1 pr2
8 88 date1 pr2
5 55 date2 pr1
6 66 date2 pr1
7 77 date2 pr2
8 88 date2 pr2
and we can also check that data all located in exceped directionary
----------------------------------------------------------------------
the script to create partition_test_input_spark2 and
create table partition_test_input_spark2
(member_id2 string,
name2 string,
stat_date2 string,
province2 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
LOAD DATA LOCAL INPATH '/root/Desktop/testpartition.txt' OVERWRITE INTO TABLE partition_test_input_spark2;
()
create table partition_test_spark
(member_id string,
name string
)
partitioned by (
stat_date string,
province string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
> Add "Dynamic Partition" support to Spark Sql hive
> ---------------------------------------------------
>
> Key: SPARK-3007
> URL: https://issues.apache.org/jira/browse/SPARK-3007
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: baishuo
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org