You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Prasad Chakka (JIRA)" <ji...@apache.org> on 2009/05/13 22:56:45 UTC
[jira] Created: (HIVE-488) loading into a partition with more than
one partition column fails if the partition is not created before.
loading into a partition with more than one partition column fails if the partition is not created before.
----------------------------------------------------------------------------------------------------------
Key: HIVE-488
URL: https://issues.apache.org/jira/browse/HIVE-488
Project: Hadoop Hive
Issue Type: Bug
Reporter: Prasad Chakka
Assignee: Prasad Chakka
Following test fails on HDFS cluster but not on local file system.
drop table hive_test_src;
drop table hive_test_dst;
create table hive_test_src ( col1 string ) stored as textfile ;
load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
drop table hive_test_src;
drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-488) loading into a partition with more
than one partition column fails if the partition is not created before.
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710573#action_12710573 ]
Ashish Thusoo commented on HIVE-488:
------------------------------------
Looks like this is a JIRA bug.
+1 for the fix.
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-488) loading into a partition with more than
one partition column fails if the partition is not created before.
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Chakka updated HIVE-488:
-------------------------------
Attachment: hive-488.patch
if the partition is to be copied to a/b/c and if a/b doesn't exist, rename(tmpdir, a/b/c) fails. this failure case was never caught but with HIVE-442 this bug is exposed since HIVE-442 explicitly does not create the partition before data is moved. this patch tries to create parent directory (a/b) so it preserves the semantics of not creating a partition until data is moved.
also removed 'add partition' since load automatically creates the partition if one doesn't exist.
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-488) loading into a partition with more than
one partition column fails if the partition is not created before.
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Chakka updated HIVE-488:
-------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Fix For: 0.4.0
>
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-488) loading into a partition with more
than one partition column fails if the partition is not created before.
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712887#action_12712887 ]
Joydeep Sen Sarma commented on HIVE-488:
----------------------------------------
this looks like it's been committed for a while .. - mark as fixed?
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-488) loading into a partition with more than
one partition column fails if the partition is not created before.
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Chakka updated HIVE-488:
-------------------------------
Status: Patch Available (was: Open)
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-488) loading into a partition with more
than one partition column fails if the partition is not created before.
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710563#action_12710563 ]
Ashish Thusoo commented on HIVE-488:
------------------------------------
I think the checkin was not meant for HIVE-488.
Also, how will the code work for the case where the partition hierarchy is n deep where n >= 3 ?
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-488) loading into a partition with more than
one partition column fails if the partition is not created before.
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Chakka updated HIVE-488:
-------------------------------
Attachment: hive-488.2.patch
added the data file used in the test.
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
> Attachments: hive-488.2.patch, hive-488.patch
>
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-488) loading into a partition with more than
one partition column fails if the partition is not created before.
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Chakka updated HIVE-488:
-------------------------------
Component/s: Query Processor
Affects Version/s: 0.4.0
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-488) loading into a partition with more
than one partition column fails if the partition is not created before.
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709224#action_12709224 ]
Prasad Chakka commented on HIVE-488:
------------------------------------
a simple work around is to create the partition before doing insert overwrite using query like below
alter table hive_test_dst add partition( pcol1='test_part', pcol2='test_part');
> loading into a partition with more than one partition column fails if the partition is not created before.
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-488
> URL: https://issues.apache.org/jira/browse/HIVE-488
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Prasad Chakka
> Assignee: Prasad Chakka
>
> Following test fails on HDFS cluster but not on local file system.
> drop table hive_test_src;
> drop table hive_test_dst;
> create table hive_test_src ( col1 string ) stored as textfile ;
> load data local inpath '../data/files/test.dat' overwrite into table hive_test_src ;
> create table hive_test_dst ( col1 string ) partitioned by ( pcol1 string , pcol2 string) stored as sequencefile;
> insert overwrite table hive_test_dst partition ( pcol1='test_part', pcol2='test_part') select col1 from hive_test_src ;
> select * from hive_test_dst where pcol1='test_part' and pcol2='test_part'; -- returns zero rows.
> drop table hive_test_src;
> drop table hive_test_dst;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.