You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Chaoyu Tang <ct...@gmail.com> on 2014/08/05 03:12:11 UTC
Review Request 24284: HIVE-7441:Custom partition scheme gets rewritten with
hive scheme upon concatenate
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24284/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-7441
https://issues.apache.org/jira/browse/HIVE-7441
Repository: hive-git
Description
-------
Problem (See HIVE-7441 description):
When Hive tries to concatenate the partition data files with hdfs cache is disabled (fs.hdfs.impl.disable.cache=true), the concatenated data file will be created under a partition directory with a name in form of /part=value (e.g. /b=part1). If the original partition directory is different (e.g /part1), querying the concatenated partition returns empty result since the partition location metastore still points to the original partition directory (/part1).
Cause:
When hdfs cache is disabled, filesystem equals comparison returns false for two fs objects even from same filesystem, which causes the original partition directory (/part1) NOT to be assigned and used as the concatenated partition destination directory.
Solution:
Before HDFS-4321 is fixed, we compare two filesystems using their uri.
Diffs
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java 8dcd4cf4432b261dff8d549b07704d75f4f1c769
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 3126880aa1a1c5ed93c9576e84c980511e86145b
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a7e50adf02e795d192a29d918e3ac24665461e06
ql/src/test/queries/clientpositive/alter_merge_3.q PRE-CREATION
ql/src/test/results/clientpositive/alter_merge_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/24284/diff/
Testing
-------
1. Manual tests verified the solution fixed the reported issue
2. New added qtests passed
3. Submit the patch and run pre-commit tests to see if there is any regression.
Thanks,
Chaoyu Tang
Re: Review Request 24284: HIVE-7441:Custom partition scheme gets rewritten
with hive scheme upon concatenate
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24284/#review49631
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
<https://reviews.apache.org/r/24284/#comment86881>
Can we get rid of the space after the '!'?
- Szehon Ho
On Aug. 5, 2014, 1:12 a.m., Chaoyu Tang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24284/
> -----------------------------------------------------------
>
> (Updated Aug. 5, 2014, 1:12 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7441
> https://issues.apache.org/jira/browse/HIVE-7441
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Problem (See HIVE-7441 description):
> When Hive tries to concatenate the partition data files with hdfs cache is disabled (fs.hdfs.impl.disable.cache=true), the concatenated data file will be created under a partition directory with a name in form of /part=value (e.g. /b=part1). If the original partition directory is different (e.g /part1), querying the concatenated partition returns empty result since the partition location metastore still points to the original partition directory (/part1).
>
> Cause:
> When hdfs cache is disabled, filesystem equals comparison returns false for two fs objects even from same filesystem, which causes the original partition directory (/part1) NOT to be assigned and used as the concatenated partition destination directory.
>
> Solution:
> Before HDFS-4321 is fixed, we compare two filesystems using their uri.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java 8dcd4cf4432b261dff8d549b07704d75f4f1c769
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 3126880aa1a1c5ed93c9576e84c980511e86145b
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a7e50adf02e795d192a29d918e3ac24665461e06
> ql/src/test/queries/clientpositive/alter_merge_3.q PRE-CREATION
> ql/src/test/results/clientpositive/alter_merge_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24284/diff/
>
>
> Testing
> -------
>
> 1. Manual tests verified the solution fixed the reported issue
> 2. New added qtests passed
> 3. Submit the patch and run pre-commit tests to see if there is any regression.
>
>
> Thanks,
>
> Chaoyu Tang
>
>
Re: Review Request 24284: HIVE-7441:Custom partition scheme gets rewritten
with hive scheme upon concatenate
Posted by Chaoyu Tang <ct...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24284/
-----------------------------------------------------------
(Updated Aug. 5, 2014, 10 p.m.)
Review request for hive.
Changes
-------
Thanks, Szehon, for the review. I made the changes based on the feedback:
1. Remove the space after !
2. Fixed the failed tests. I believe the test failures were from the order of the selected rows. I changed the test to query row count since we only want to ensure no empty result is returned after concatenate in the test.
Bugs: HIVE-7441
https://issues.apache.org/jira/browse/HIVE-7441
Repository: hive-git
Description
-------
Problem (See HIVE-7441 description):
When Hive tries to concatenate the partition data files with hdfs cache is disabled (fs.hdfs.impl.disable.cache=true), the concatenated data file will be created under a partition directory with a name in form of /part=value (e.g. /b=part1). If the original partition directory is different (e.g /part1), querying the concatenated partition returns empty result since the partition location metastore still points to the original partition directory (/part1).
Cause:
When hdfs cache is disabled, filesystem equals comparison returns false for two fs objects even from same filesystem, which causes the original partition directory (/part1) NOT to be assigned and used as the concatenated partition destination directory.
Solution:
Before HDFS-4321 is fixed, we compare two filesystems using their uri.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java 8dcd4cf4432b261dff8d549b07704d75f4f1c769
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 3126880aa1a1c5ed93c9576e84c980511e86145b
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a7e50adf02e795d192a29d918e3ac24665461e06
ql/src/test/queries/clientpositive/alter_merge_3.q PRE-CREATION
ql/src/test/results/clientpositive/alter_merge_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/24284/diff/
Testing
-------
1. Manual tests verified the solution fixed the reported issue
2. New added qtests passed
3. Submit the patch and run pre-commit tests to see if there is any regression.
Thanks,
Chaoyu Tang