You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (Jira)" <ji...@apache.org> on 2019/08/29 09:31:00 UTC
[jira] [Commented] (HIVE-22088) Dynamic partition insert problem on
table with "=" in location path spec
[ https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918442#comment-16918442 ]
Hive QA commented on HIVE-22088:
--------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 28s{color} | {color:blue} standalone-metastore/metastore-common in master has 32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 52s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 35s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s{color} | {color:red} ql: The patch generated 1 new + 357 unchanged - 0 fixed = 358 total (was 357) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 53s{color} | {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18415/dev-support/hive-personality.sh |
| git revision | master / d26516e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18415/yetus/diff-checkstyle-ql.txt |
| modules | C: standalone-metastore/metastore-common ql hcatalog/core U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18415/yetus.txt |
| Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
> Dynamic partition insert problem on table with "=" in location path spec
> ------------------------------------------------------------------------
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.0.0, 2.3.4
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> OS: Ubuntu 16.04.4 LTS
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> Reporter: Puneet Khatod
> Assignee: Hui An
> Priority: Major
> Attachments: HIVE-22088.patch.1, HIVE-22088.patch.2
>
>
> If external table location spec has a '=' sign (coincidentally partition specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different external tables on top of the data. Many times the tables have different partition depths depending on how data is organized.
> Like in below example, there are individual customer specific tables and queries/jobs to insert data partitioned by type. And there is another table to give the consolidated data view of all the customers, thus have two level partition customer and type.
> The job to insert customer specific data into customer specific table fails if we use dynamic partitioning. Static partition insert on same table works fine though.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#ff0000}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains non-partition columns
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are under the defined table root/base path. If the path itself has partition style format (customer=abc in above example) then that should not be considered as partition as it is outside the scope of the table.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)